Merge pull request #66 from 93-infinity/master

Modified & enhanced docs wrt paths

README.md

@@ -43,13 +43,14 @@ Download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhosty
 
 ## Installation (Windows):
 
-Because Windows has a path length limit of 260 characters, and because of the work required (and the fact that the Windows version was untested when it existed), Windows users needs to use the scripts in WSL (or something linux-on-windows). WSL is the only thing that bypass the path length limit so every video will get downloaded with it. However when playing those videos with a native Windows software the path length limit might be an issue, I can't do anything about that.
-
-Installation using WSL (recommended): TODO
-
+The Windows version, while it existed, required a lot of work to maintain because it needed specific changes compared to the linux version, and was never extensively tested. Windows users need to use the scripts in WSL, Cygwin, Git Bash, or some other Linux-on-Windows environment that enables Bash functionality in Windows.
+ 
+Installation using WSL (recommended): [Here](docs/WSL.md)
 
 Installation using Cygwin, Git Bash or similar: [Here](docs/Cygwin-Git-Bash.md)
 
+Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds.
+
 ---
 
 ## Usage: 

docs/About-Paths.md

@@ -0,0 +1,74 @@
+## About Paths
+
+### What is a path?
+
+The path of a file or folder specifies its location on the filesystem. The path length refers to the total number of character in the path.
+- C:\Users\username\Pictures\Places\France\Paris.jpeg (a file path, 51 characters)
+- C:\Program Files (a folder path, 16 characters)
+
+---
+
+### Path length limits on Windows
+
+Windows has a 260-character path length limit for compatibility reasons. This means Windows and other programs will refuse to:
+- create files or folders that exceed this limit,
+- rename existing files or folders to exceed this limit.
+
+It's possible for programs to use alternate methods to bypass the 260-character limit. This is because Windows' underlying NTFS filesystem supports "long paths" with lengths of up to 32,767 characters, [even since the days of Windows XP](https://stackoverflow.com/questions/265769/maximum-filename-length-in-ntfs-windows-xp-and-windows-vista). Incidentally, this is even higher than Linux's 4096-character limit.
+
+However files and folders created by programs that support long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. They will not be able to be easily opened, copied, moved, renamed, or deleted, without the use of special tools. The files themselves are perfectly valid and not corrupt, and having long paths is not a problem per se, just an inconvenience.
+
+---
+
+### Path lengths and the scripts
+
+These scripts were designed to both neatly organize videos as well as preserve as much information about them as possible, since this information is often valuable to archivists. The names of both the videos and the folders they are stored in may include (in various combinations):
+- channel name,
+- playlist name,
+- upload date,
+- full video title,
+- YouTube video ID.
+
+Practically speaking, this means videos downloaded with these scripts will often exceed Windows' 260-character limit. Hence, you should keep the path as short as possible; store the scripts in the root (top level) of the drive. Place them in "C:\\" or "D:\\" instead of "C:\Users\username\Downloads\". Some videos will likely still exceed the 260-character limit, but this is good practice and should be done regardless.
+
+---
+
+### Play video files with long paths
+
+"VLC Media Player" downloaded from the VLC website [does not support long paths](https://forum.videolan.org/viewtopic.php?t=150542). Videos cannot be opened.
+
+Other programs may work. (WIP)
+
+"Metro" apps appear to support long paths. "Movies & TV" (built in to Windows 10), or VLC (from the Microsoft Store).
+
+Video players running *inside* WSL do work. To get the videos to display, you need to set up an Xserver (e.g. Xming), or use MobaXterm which has one built in, and possibly install PulseAudio (untested). This is a somewhat involved process and is not recommended for beginners.
+
+---
+
+### Manipulate (copy, move, rename, delete) files with long paths
+
+Files and folders with long paths will be impossible to manipulate in programs that don't support them, or in Windows itself. Overcome this by manipulating files inside WSL:
+* Download a file manager like mc (Midnight Commander) - recommended
+* Use built-in Linux commandline tools like cp (to copy), mv (to move), and rm (to delete)
+
+[HardLinkShell](https://schinagl.priv.at/nt/hardlinkshellext/linkshellextension.html) (Windows) enables copying or moving files with long paths.
+
+---
+
+### Other workarounds
+
+Enable a "Group Policy" setting introduced in recent versions of Windows 10 to [remove the 260-character limit](https://superuser.com/questions/1119883/windows-10-enable-ntfs-long-paths-policy-option-missing) and enable support for long paths. For now it doesn't do much; applications need to be updated to support this setting (not even Windows Explorer seems to).
+
+Choose a shorter path by not saving files too deeply into the folder tree. This is not really an option for these scripts, if you already store them in the root of the drive.
+
+Organize videos and their containing folders with your own organization system (i.e., shorten their names). This is an issue of personal preference. If you break the scripts because they can't find files where it expects them to be, that is your problem to deal with.
+
+---
+
+### Further resources 
+
+[PathLengthChecker](https://github.com/deadlydog/PathLengthChecker) - Find out the path length of files/folders
+
+[Further reading on WSL and Path Lengths](https://channel9.msdn.com/Blogs/Seth-Juarez/Windows-Subsystem-for-Linux-File-System)
+
+[Background on Maximum Path Length Limitations](https://docs.microsoft.com/en-us/windows/win32/fileio/maximum-file-path-limitation)

docs/Cygwin-Git-Bash.md

@@ -1,10 +1,11 @@
-### Installation using Cygwin, Git Bash or similar:
+## Installation using Cygwin, Git Bash or similar
 
-#### Known Issues
-* Windows has a hard limit of 260 characters for the filepath. This can cause issues with video downloading. It is recommended that you make your filepath as short as possible.
+### Known Issues
 * If you are receiving a permission denied error when attempting to execute python, [this thread](https://stackoverflow.com/questions/56974927/permission-denied-trying-to-run-python-on-windows-10) may be helpful.
+* Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds.
 
-#### Required Software
+
+### Required Software
 In order to run these scripts, the following software packages must be installed:
 
 * [Cygwin](https://www.cygwin.com/), [Git Bash](http://git-scm.com), or some other application that enables Bash functionality in Windows.
@@ -13,7 +14,7 @@ In order to run these scripts, the following software packages must be installed
 * [ffmpeg](https://www.ffmpeg.org/) - Make sure this is in your PATH.
 * [atomicparsley](https://github.com/wez/atomicparsley) (optional) - Make sure this is in your PATH.
 
-#### Configuration
+### Configuration
 First, make sure that all dependencies are installed and your PATH is configured properly by opening your Bash terminal of choice and executing the following:
 
 * `python --version`

docs/Details.md

@@ -1,4 +1,4 @@
-# Details:
+## Details:
 
 `--format "(bestvideo[vcodec^=av01][height>=4320][fps>30]/bestvideo[vcodec^=vp9.2][height>=4320][fps>30]/bestvideo[vcodec^=vp9][height>=4320][fps>30]/bestvideo[vcodec^=avc1][height>=4320][fps>30]/bestvideo[height>=4320][fps>30]/bestvideo[vcodec^=av01][height>=4320]/bestvideo[vcodec^=vp9.2][height>=4320]/bestvideo[vcodec^=vp9][height>=4320]/bestvideo[vcodec^=avc1][height>=4320]/bestvideo[height>=4320]/bestvideo[vcodec^=av01][height>=2880][fps>30]/bestvideo[vcodec^=vp9.2][height>=2880][fps>30]/bestvideo[vcodec^=vp9][height>=2880][fps>30]/bestvideo[vcodec^=avc1][height>=2880][fps>30]/bestvideo[height>=2880][fps>30]/bestvideo[vcodec^=av01][height>=2880]/bestvideo[vcodec^=vp9.2][height>=2880]/bestvideo[vcodec^=vp9][height>=2880]/bestvideo[vcodec^=avc1][height>=2880]/bestvideo[height>=2880]/bestvideo[vcodec^=av01][height>=2160][fps>30]/bestvideo[vcodec^=vp9.2][height>=2160][fps>30]/bestvideo[vcodec^=vp9][height>=2160][fps>30]/bestvideo[vcodec^=avc1][height>=2160][fps>30]/bestvideo[height>=2160][fps>30]/bestvideo[vcodec^=av01][height>=2160]/bestvideo[vcodec^=vp9.2][height>=2160]/bestvideo[vcodec^=vp9][height>=2160]/bestvideo[vcodec^=avc1][height>=2160]/bestvideo[height>=2160]/bestvideo[vcodec^=av01][height>=1440][fps>30]/bestvideo[vcodec^=vp9.2][height>=1440][fps>30]/bestvideo[vcodec^=vp9][height>=1440][fps>30]/bestvideo[vcodec^=avc1][height>=1440][fps>30]/bestvideo[height>=1440][fps>30]/bestvideo[vcodec^=av01][height>=1440]/bestvideo[vcodec^=vp9.2][height>=1440]/bestvideo[vcodec^=vp9][height>=1440]/bestvideo[vcodec^=avc1][height>=1440]/bestvideo[height>=1440]/bestvideo[vcodec^=av01][height>=1080][fps>30]/bestvideo[vcodec^=vp9.2][height>=1080][fps>30]/bestvideo[vcodec^=vp9][height>=1080][fps>30]/bestvideo[vcodec^=avc1][height>=1080][fps>30]/bestvideo[height>=1080][fps>30]/bestvideo[vcodec^=av01][height>=1080]/bestvideo[vcodec^=vp9.2][height>=1080]/bestvideo[vcodec^=vp9][height>=1080]/bestvideo[vcodec^=avc1][height>=1080]/bestvideo[height>=1080]/bestvideo[vcodec^=av01][height>=720][fps>30]/bestvideo[vcodec^=vp9.2][height>=720][fps>30]/bestvideo[vcodec^=vp9][height>=720][fps>30]/bestvideo[vcodec^=avc1][height>=720][fps>30]/bestvideo[height>=720][fps>30]/bestvideo[vcodec^=av01][height>=720]/bestvideo[vcodec^=vp9.2][height>=720]/bestvideo[vcodec^=vp9][height>=720]/bestvideo[vcodec^=avc1][height>=720]/bestvideo[height>=720]/bestvideo[vcodec^=av01][height>=480][fps>30]/bestvideo[vcodec^=vp9.2][height>=480][fps>30]/bestvideo[vcodec^=vp9][height>=480][fps>30]/bestvideo[vcodec^=avc1][height>=480][fps>30]/bestvideo[height>=480][fps>30]/bestvideo[vcodec^=av01][height>=480]/bestvideo[vcodec^=vp9.2][height>=480]/bestvideo[vcodec^=vp9][height>=480]/bestvideo[vcodec^=avc1][height>=480]/bestvideo[height>=480]/bestvideo[vcodec^=av01][height>=360][fps>30]/bestvideo[vcodec^=vp9.2][height>=360][fps>30]/bestvideo[vcodec^=vp9][height>=360][fps>30]/bestvideo[vcodec^=avc1][height>=360][fps>30]/bestvideo[height>=360][fps>30]/bestvideo[vcodec^=av01][height>=360]/bestvideo[vcodec^=vp9.2][height>=360]/bestvideo[vcodec^=vp9][height>=360]/bestvideo[vcodec^=avc1][height>=360]/bestvideo[height>=360]/bestvideo[vcodec^=avc1][height>=240][fps>30]/bestvideo[vcodec^=av01][height>=240][fps>30]/bestvideo[vcodec^=vp9.2][height>=240][fps>30]/bestvideo[vcodec^=vp9][height>=240][fps>30]/bestvideo[height>=240][fps>30]/bestvideo[vcodec^=avc1][height>=240]/bestvideo[vcodec^=av01][height>=240]/bestvideo[vcodec^=vp9.2][height>=240]/bestvideo[vcodec^=vp9][height>=240]/bestvideo[height>=240]/bestvideo[vcodec^=avc1][height>=144][fps>30]/bestvideo[vcodec^=av01][height>=144][fps>30]/bestvideo[vcodec^=vp9.2][height>=144][fps>30]/bestvideo[vcodec^=vp9][height>=144][fps>30]/bestvideo[height>=144][fps>30]/bestvideo[vcodec^=avc1][height>=144]/bestvideo[vcodec^=av01][height>=144]/bestvideo[vcodec^=vp9.2][height>=144]/bestvideo[vcodec^=vp9][height>=144]/bestvideo[height>=144]/bestvideo)+(bestaudio[acodec^=opus]/bestaudio)/best"`: Tell yt-dlp to download the best quality available prioritizing the most compressed/recent codecs, except for 240p and 144p where AVC is prioritized (Original idea by [Veloldo](https://redd.it/c6fh4x))
 

docs/WSL.md

@@ -0,0 +1,17 @@
+## Installation using WSL
+WSL allows the running of a modified Linux environment (Ubuntu, Debian, OpenSUSE, and more) inside Windows 10. Unlike a virtual machine, it can directly interact with Windows files on the hard drive. 
+
+Main benefit: it's created by Microsoft, so it's more likely to be better supported. Benefits for the purposes of these scripts are the ability to keep dependencies (ffmpeg, atomicparsley, yt-dlp) up-to-date with package managers like apt or pip. 
+
+However the setup process does necessitate a relatively large one-time download and storing of WSL's files (approximately 2GB) - users taking issue with this may consider the other options such as Cygwin or Git Bash. [Compare the above here.](https://askubuntu.com/questions/1042285/reduce-size-of-a-wsl-installation-ubuntu-18-on-windows-10)
+
+
+### Known Issues
+* The latest version of WSL (WSL2) stores the Linux filesystem hierarchy ( /* ) inside a virtual hard disk file. To avoid this file ballooning in size ([and having to manually shrink it in the future](https://stephenreescarter.net/how-to-shrink-a-wsl2-virtual-disk/)), it's recommended to store the scripts and videos outside of the Linux filesystem, inside a Windows drive (e.g. "/mnt/c/" for "C:\\", or "/mnt/d/" for a drive mounted at "D:\\" )
+* Downloaded videos will likely exceed Windows' 260-character path length limit and not be playable out of the box. You should read more about [paths](docs/About-Paths.md) to familiarize yourself with the issue, as well as some potential workarounds.
+
+
+### Required Software
+Follow the "Manual Install" instructions in the [Windows Subsystem for Linux Installation Guide for Windows 10](https://docs.microsoft.com/en-us/windows/wsl/install-win10). 
+
+Once WSL is installed, download the latest [release](https://github.com/TheFrenchGhosty/TheFrenchGhostys-YouTube-DL-Archivist-Scripts/releases) and follow the "Installation (Linux)" section of this README.