Improve plugin architecture (#5553)

to make plugins easier to develop and use:
* Plugins are now loaded as namespace packages.
* Plugins can be loaded in any distribution of yt-dlp (binary, pip, source, etc.).
* Plugin packages can be installed and managed via pip, or dropped into any of the documented locations.
* Users do not need to edit any code files to install plugins.
* Backwards-compatible with previous plugin architecture.

As a side-effect, yt-dlp will now search in a few more locations for config files.

Closes https://github.com/yt-dlp/yt-dlp/issues/1389

Authored by: flashdagger, coletdjnz, pukkandan, Grub4K
Co-authored-by: Marcel <flashdagger@googlemail.com>
Co-authored-by: pukkandan <pukkandan.ytdlp@gmail.com>
Co-authored-by: Simon Sawicki <accounts@grub4k.xyz>

.gitignore

@@ -120,9 +120,5 @@ yt-dlp.zip
 */extractor/lazy_extractors.py
 
 # Plugins
-ytdlp_plugins/extractor/*
-!ytdlp_plugins/extractor/__init__.py
-!ytdlp_plugins/extractor/sample.py
-ytdlp_plugins/postprocessor/*
-!ytdlp_plugins/postprocessor/__init__.py
-!ytdlp_plugins/postprocessor/sample.py
+ytdlp_plugins/*
+yt-dlp-plugins/*

README.md

@@ -61,6 +61,8 @@ yt-dlp is a [youtube-dl](https://github.com/ytdl-org/youtube-dl) fork based on t
     * [Modifying metadata examples](#modifying-metadata-examples)
 * [EXTRACTOR ARGUMENTS](#extractor-arguments)
 * [PLUGINS](#plugins)
+    * [Installing Plugins](#installing-plugins)
+    * [Developing Plugins](#developing-plugins)
 * [EMBEDDING YT-DLP](#embedding-yt-dlp)
     * [Embedding examples](#embedding-examples)
 * [DEPRECATED OPTIONS](#deprecated-options)
@@ -1110,15 +1112,20 @@ You can configure yt-dlp by placing any supported command line option to a confi
     * If `-P` is not given, the current directory is searched
 1. **User Configuration**:
     * `${XDG_CONFIG_HOME}/yt-dlp/config` (recommended on Linux/macOS)
+    * `${XDG_CONFIG_HOME}/yt-dlp/config.txt`
     * `${XDG_CONFIG_HOME}/yt-dlp.conf`
     * `${APPDATA}/yt-dlp/config` (recommended on Windows)
     * `${APPDATA}/yt-dlp/config.txt`
     * `~/yt-dlp.conf`
     * `~/yt-dlp.conf.txt`
+    * `~/.yt-dlp/config`
+    * `~/.yt-dlp/config.txt`
 
     See also: [Notes about environment variables](#notes-about-environment-variables)
 1. **System Configuration**:
     * `/etc/yt-dlp.conf`
+    * `/etc/yt-dlp/config`
+    * `/etc/yt-dlp/config.txt`
 
 E.g. with the following configuration file yt-dlp will always extract the audio, not copy the mtime, use a proxy and save all videos under `YouTube` directory in your home directory:
 ```
@@ -1789,19 +1796,68 @@ NOTE: These options may be changed/removed in the future without concern for bac
 
 # PLUGINS
 
-Plugins are loaded from `<root-dir>/ytdlp_plugins/<type>/__init__.py`; where `<root-dir>` is the directory of the binary (`<root-dir>/yt-dlp`), or the root directory of the module if you are running directly from source-code (`<root dir>/yt_dlp/__main__.py`). Plugins are currently not supported for the `pip` version
+Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. **Use plugins at your own risk and only if you trust the code!**
 
-Plugins can be of `<type>`s `extractor` or `postprocessor`. Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it. Postprocessor plugins can be invoked using `--use-postprocessor NAME`.
+Plugins can be of `<type>`s `extractor` or `postprocessor`. 
+- Extractor plugins do not need to be enabled from the CLI and are automatically invoked when the input URL is suitable for it. 
+- Extractor plugins take priority over builtin extractors.
+- Postprocessor plugins can be invoked using `--use-postprocessor NAME`.
 
-See [ytdlp_plugins](ytdlp_plugins) for example plugins.
 
-Note that **all** plugins are imported even if not invoked, and that **there are no checks** performed on plugin code. Use plugins at your own risk and only if you trust the code
+Plugins are loaded from the namespace packages `yt_dlp_plugins.extractor` and `yt_dlp_plugins.postprocessor`.
 
-If you are a plugin author, add [ytdlp-plugins](https://github.com/topics/ytdlp-plugins) as a topic to your repository for discoverability
+In other words, the file structure on the disk looks something like:
+    
+        yt_dlp_plugins/
+            extractor/
+                myplugin.py
+            postprocessor/
+                myplugin.py
+
+yt-dlp looks for these `yt_dlp_plugins` namespace folders in many locations (see below) and loads in plugins from **all** of them.
 
 See the [wiki for some known plugins](https://github.com/yt-dlp/yt-dlp/wiki/Plugins)
 
+## Installing Plugins
 
+Plugins can be installed using various methods and locations.
+
+1. **Configuration directories**:
+   Plugin packages (containing a `yt_dlp_plugins` namespace folder) can be dropped into the following standard [configuration locations](#configuration):
+    * **User Plugins**
+      * `${XDG_CONFIG_HOME}/yt-dlp/plugins/<package name>/yt_dlp_plugins/` (recommended on Linux/macOS)
+      * `${XDG_CONFIG_HOME}/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
+      * `${APPDATA}/yt-dlp/plugins/<package name>/yt_dlp_plugins/` (recommended on Windows)
+      * `~/.yt-dlp/plugins/<package name>/yt_dlp_plugins/`
+      * `~/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
+    * **System Plugins**
+      * `/etc/yt-dlp/plugins/<package name>/yt_dlp_plugins/`
+      * `/etc/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
+2. **Executable location**: Plugin packages can similarly be installed in a `yt-dlp-plugins` directory under the executable location:
+    * Binary: where `<root-dir>/yt-dlp.exe`, `<root-dir>/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
+    * Source: where `<root-dir>/yt_dlp/__main__.py`, `<root-dir>/yt-dlp-plugins/<package name>/yt_dlp_plugins/`
+
+3. **pip and other locations in `PYTHONPATH`**
+    * Plugin packages can be installed and managed using `pip`. See [ytdlp-sample-plugins](https://github.com/yt-dlp/yt-dlp-sample-plugins) for an example.
+      * Note: plugin files between plugin packages installed with pip must have unique filenames
+    * Any path in `PYTHONPATH` is searched in for the `yt_dlp_plugins` namespace folder.
+      * Note: This does not apply for Pyinstaller/py2exe builds.
+
+
+.zip, .egg and .whl archives containing a `yt_dlp_plugins` namespace folder in their root are also supported. These can be placed in the same locations `yt_dlp_plugins` namespace folders can be found.
+- e.g. `${XDG_CONFIG_HOME}/yt-dlp/plugins/mypluginpkg.zip` where `mypluginpkg.zip` contains `yt_dlp_plugins/<type>/myplugin.py`
+
+Run yt-dlp with `--verbose`/`-v` to check if the plugin has been loaded.
+
+## Developing Plugins
+
+See [ytdlp-sample-plugins](https://github.com/yt-dlp/yt-dlp-sample-plugins) for a sample plugin package with instructions on how to set up an environment for plugin development. 
+
+All public classes with a name ending in `IE` are imported from each file. This respects underscore prefix (e.g. `_MyBasePluginIE` is private) and `__all__`. Modules can similarly be excluded by prefixing the module name with an underscore (e.g. `_myplugin.py`)
+
+If you are a plugin author, add [yt-dlp-plugins](https://github.com/topics/yt-dlp-plugins) as a topic to your repository for discoverability
+
+See the [Developer Instructions](https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#developer-instructions) on how to write and test an extractor.
 
 # EMBEDDING YT-DLP
 

devscripts/make_lazy_extractors.py

@@ -40,8 +40,12 @@ def main():
 
     _ALL_CLASSES = get_all_ies()  # Must be before import
 
+    import yt_dlp.plugins
     from yt_dlp.extractor.common import InfoExtractor, SearchInfoExtractor
 
+    # Filter out plugins
+    _ALL_CLASSES = [cls for cls in _ALL_CLASSES if not cls.__module__.startswith(f'{yt_dlp.plugins.PACKAGE_NAME}.')]
+
     DummyInfoExtractor = type('InfoExtractor', (InfoExtractor,), {'IE_NAME': NO_ATTR})
     module_src = '\n'.join((
         MODULE_TEMPLATE,

test/test_plugins.py

@@ -0,0 +1,73 @@
+import importlib
+import os
+import shutil
+import sys
+import unittest
+from pathlib import Path
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+TEST_DATA_DIR = Path(os.path.dirname(os.path.abspath(__file__)), 'testdata')
+sys.path.append(str(TEST_DATA_DIR))
+importlib.invalidate_caches()
+
+from yt_dlp.plugins import PACKAGE_NAME, directories, load_plugins
+
+
+class TestPlugins(unittest.TestCase):
+
+    TEST_PLUGIN_DIR = TEST_DATA_DIR / PACKAGE_NAME
+
+    def test_directories_containing_plugins(self):
+        self.assertIn(self.TEST_PLUGIN_DIR, map(Path, directories()))
+
+    def test_extractor_classes(self):
+        for module_name in tuple(sys.modules):
+            if module_name.startswith(f'{PACKAGE_NAME}.extractor'):
+                del sys.modules[module_name]
+        plugins_ie = load_plugins('extractor', 'IE')
+
+        self.assertIn(f'{PACKAGE_NAME}.extractor.normal', sys.modules.keys())
+        self.assertIn('NormalPluginIE', plugins_ie.keys())
+
+        # don't load modules with underscore prefix
+        self.assertFalse(
+            f'{PACKAGE_NAME}.extractor._ignore' in sys.modules.keys(),
+            'loaded module beginning with underscore')
+        self.assertNotIn('IgnorePluginIE', plugins_ie.keys())
+
+        # Don't load extractors with underscore prefix
+        self.assertNotIn('_IgnoreUnderscorePluginIE', plugins_ie.keys())
+
+        # Don't load extractors not specified in __all__ (if supplied)
+        self.assertNotIn('IgnoreNotInAllPluginIE', plugins_ie.keys())
+        self.assertIn('InAllPluginIE', plugins_ie.keys())
+
+    def test_postprocessor_classes(self):
+        plugins_pp = load_plugins('postprocessor', 'PP')
+        self.assertIn('NormalPluginPP', plugins_pp.keys())
+
+    def test_importing_zipped_module(self):
+        zip_path = TEST_DATA_DIR / 'zipped_plugins.zip'
+        shutil.make_archive(str(zip_path)[:-4], 'zip', str(zip_path)[:-4])
+        sys.path.append(str(zip_path))  # add zip to search paths
+        importlib.invalidate_caches()  # reset the import caches
+
+        try:
+            for plugin_type in ('extractor', 'postprocessor'):
+                package = importlib.import_module(f'{PACKAGE_NAME}.{plugin_type}')
+                self.assertIn(zip_path / PACKAGE_NAME / plugin_type, map(Path, package.__path__))
+
+            plugins_ie = load_plugins('extractor', 'IE')
+            self.assertIn('ZippedPluginIE', plugins_ie.keys())
+
+            plugins_pp = load_plugins('postprocessor', 'PP')
+            self.assertIn('ZippedPluginPP', plugins_pp.keys())
+
+        finally:
+            sys.path.remove(str(zip_path))
+            os.remove(zip_path)
+            importlib.invalidate_caches()  # reset the import caches
+
+
+if __name__ == '__main__':
+    unittest.main()

test/testdata/yt_dlp_plugins/extractor/_ignore.py

@@ -0,0 +1,5 @@
+from yt_dlp.extractor.common import InfoExtractor
+
+
+class IgnorePluginIE(InfoExtractor):
+    pass

test/testdata/yt_dlp_plugins/extractor/ignore.py

@@ -0,0 +1,12 @@
+from yt_dlp.extractor.common import InfoExtractor
+
+
+class IgnoreNotInAllPluginIE(InfoExtractor):
+    pass
+
+
+class InAllPluginIE(InfoExtractor):
+    pass
+
+
+__all__ = ['InAllPluginIE']

test/testdata/yt_dlp_plugins/extractor/normal.py

@@ -0,0 +1,9 @@
+from yt_dlp.extractor.common import InfoExtractor
+
+
+class NormalPluginIE(InfoExtractor):
+    pass
+
+
+class _IgnoreUnderscorePluginIE(InfoExtractor):
+    pass

test/testdata/yt_dlp_plugins/postprocessor/normal.py

@@ -0,0 +1,5 @@
+from yt_dlp.postprocessor.common import PostProcessor
+
+
+class NormalPluginPP(PostProcessor):
+    pass

test/testdata/zipped_plugins/yt_dlp_plugins/extractor/zipped.py

@@ -0,0 +1,5 @@
+from yt_dlp.extractor.common import InfoExtractor
+
+
+class ZippedPluginIE(InfoExtractor):
+    pass

test/testdata/zipped_plugins/yt_dlp_plugins/postprocessor/zipped.py

@@ -0,0 +1,5 @@
+from yt_dlp.postprocessor.common import PostProcessor
+
+
+class ZippedPluginPP(PostProcessor):
+    pass

yt_dlp/YoutubeDL.py

@@ -32,6 +32,7 @@ from .extractor import gen_extractor_classes, get_info_extractor
 from .extractor.common import UnsupportedURLIE
 from .extractor.openload import PhantomJSwrapper
 from .minicurses import format_text
+from .plugins import directories as plugin_directories
 from .postprocessor import _PLUGIN_CLASSES as plugin_postprocessors
 from .postprocessor import (
     EmbedThumbnailPP,
@@ -3773,10 +3774,6 @@ class YoutubeDL:
                 write_debug('Lazy loading extractors is forcibly disabled')
             else:
                 write_debug('Lazy loading extractors is disabled')
-        if plugin_extractors or plugin_postprocessors:
-            write_debug('Plugins: %s' % [
-                '%s%s' % (klass.__name__, '' if klass.__name__ == name else f' as {name}')
-                for name, klass in itertools.chain(plugin_extractors.items(), plugin_postprocessors.items())])
         if self.params['compat_opts']:
             write_debug('Compatibility options: %s' % ', '.join(self.params['compat_opts']))
 
@@ -3810,6 +3807,16 @@ class YoutubeDL:
                 proxy_map.update(handler.proxies)
         write_debug(f'Proxy map: {proxy_map}')
 
+        for plugin_type, plugins in {'Extractor': plugin_extractors, 'Post-Processor': plugin_postprocessors}.items():
+            if not plugins:
+                continue
+            write_debug(f'{plugin_type} Plugins: %s' % (', '.join(sorted(('%s%s' % (
+                klass.__name__, '' if klass.__name__ == name else f' as {name}')
+                for name, klass in plugins.items())))))
+        plugin_dirs = plugin_directories()
+        if plugin_dirs:
+            write_debug(f'Plugin directories: {plugin_dirs}')
+
         # Not implemented
         if False and self.params.get('call_home'):
             ipaddr = self.urlopen('https://yt-dl.org/ip').read().decode()

yt_dlp/extractor/extractors.py

@@ -1,10 +1,10 @@
 import contextlib
 import os
 
-from ..utils import load_plugins
+from ..plugins import load_plugins
 
 # NB: Must be before other imports so that plugins can be correctly injected
-_PLUGIN_CLASSES = load_plugins('extractor', 'IE', {})
+_PLUGIN_CLASSES = load_plugins('extractor', 'IE')
 
 _LAZY_LOADER = False
 if not os.environ.get('YTDLP_NO_LAZY_EXTRACTORS'):

yt_dlp/options.py

@@ -29,6 +29,8 @@ from .utils import (
     expand_path,
     format_field,
     get_executable_path,
+    get_system_config_dirs,
+    get_user_config_dirs,
     join_nonempty,
     orderedSet_from_options,
     remove_end,
@@ -42,62 +44,67 @@ def parseOpts(overrideArguments=None, ignore_config_files='if_override'):
     if ignore_config_files == 'if_override':
         ignore_config_files = overrideArguments is not None
 
-    def _readUserConf(package_name, default=[]):
-        # .config
+    def _load_from_config_dirs(config_dirs):
+        for config_dir in config_dirs:
+            conf_file_path = os.path.join(config_dir, 'config')
+            conf = Config.read_file(conf_file_path, default=None)
+            if conf is None:
+                conf_file_path += '.txt'
+                conf = Config.read_file(conf_file_path, default=None)
+            if conf is not None:
+                return conf, conf_file_path
+        return None, None
+
+    def _read_user_conf(package_name, default=None):
+        # .config/package_name.conf
         xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
-        userConfFile = os.path.join(xdg_config_home, package_name, 'config')
-        if not os.path.isfile(userConfFile):
-            userConfFile = os.path.join(xdg_config_home, '%s.conf' % package_name)
-        userConf = Config.read_file(userConfFile, default=None)
-        if userConf is not None:
-            return userConf, userConfFile
+        user_conf_file = os.path.join(xdg_config_home, '%s.conf' % package_name)
+        user_conf = Config.read_file(user_conf_file, default=None)
+        if user_conf is not None:
+            return user_conf, user_conf_file
 
-        # appdata
-        appdata_dir = os.getenv('appdata')
-        if appdata_dir:
-            userConfFile = os.path.join(appdata_dir, package_name, 'config')
-            userConf = Config.read_file(userConfFile, default=None)
-            if userConf is None:
-                userConfFile += '.txt'
-                userConf = Config.read_file(userConfFile, default=None)
-        if userConf is not None:
-            return userConf, userConfFile
+        # home (~/package_name.conf or ~/package_name.conf.txt)
+        user_conf_file = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
+        user_conf = Config.read_file(user_conf_file, default=None)
+        if user_conf is None:
+            user_conf_file += '.txt'
+            user_conf = Config.read_file(user_conf_file, default=None)
+        if user_conf is not None:
+            return user_conf, user_conf_file
 
-        # home
-        userConfFile = os.path.join(compat_expanduser('~'), '%s.conf' % package_name)
-        userConf = Config.read_file(userConfFile, default=None)
-        if userConf is None:
-            userConfFile += '.txt'
-            userConf = Config.read_file(userConfFile, default=None)
-        if userConf is not None:
-            return userConf, userConfFile
+        # Package config directories (e.g. ~/.config/package_name/package_name.txt)
+        user_conf, user_conf_file = _load_from_config_dirs(get_user_config_dirs(package_name))
+        if user_conf is not None:
+            return user_conf, user_conf_file
+        return default if default is not None else [], None
 
-        return default, None
+    def _read_system_conf(package_name, default=None):
+        system_conf, system_conf_file = _load_from_config_dirs(get_system_config_dirs(package_name))
+        if system_conf is not None:
+            return system_conf, system_conf_file
+        return default if default is not None else [], None
 
-    def add_config(label, path, user=False):
+    def add_config(label, path=None, func=None):
         """ Adds config and returns whether to continue """
         if root.parse_known_args()[0].ignoreconfig:
             return False
-        # Multiple package names can be given here
-        # E.g. ('yt-dlp', 'youtube-dlc', 'youtube-dl') will look for
-        # the configuration file of any of these three packages
-        for package in ('yt-dlp',):
-            if user:
-                args, current_path = _readUserConf(package, default=None)
-            else:
-                current_path = os.path.join(path, '%s.conf' % package)
-                args = Config.read_file(current_path, default=None)
-            if args is not None:
-                root.append_config(args, current_path, label=label)
-                return True
+        elif func:
+            assert path is None
+            args, current_path = func('yt-dlp')
+        else:
+            current_path = os.path.join(path, 'yt-dlp.conf')
+            args = Config.read_file(current_path, default=None)
+        if args is not None:
+            root.append_config(args, current_path, label=label)
+            return True
         return True
 
     def load_configs():
         yield not ignore_config_files
         yield add_config('Portable', get_executable_path())
         yield add_config('Home', expand_path(root.parse_known_args()[0].paths.get('home', '')).strip())
-        yield add_config('User', None, user=True)
-        yield add_config('System', '/etc')
+        yield add_config('User', func=_read_user_conf)
+        yield add_config('System', func=_read_system_conf)
 
     opts = optparse.Values({'verbose': True, 'print_help': False})
     try:

yt_dlp/plugins.py

@@ -0,0 +1,171 @@
+import contextlib
+import importlib
+import importlib.abc
+import importlib.machinery
+import importlib.util
+import inspect
+import itertools
+import os
+import pkgutil
+import sys
+import traceback
+import zipimport
+from pathlib import Path
+from zipfile import ZipFile
+
+from .compat import functools  # isort: split
+from .compat import compat_expanduser
+from .utils import (
+    get_executable_path,
+    get_system_config_dirs,
+    get_user_config_dirs,
+    write_string,
+)
+
+PACKAGE_NAME = 'yt_dlp_plugins'
+COMPAT_PACKAGE_NAME = 'ytdlp_plugins'
+
+
+class PluginLoader(importlib.abc.Loader):
+    """Dummy loader for virtual namespace packages"""
+
+    def exec_module(self, module):
+        return None
+
+
+@functools.cache
+def dirs_in_zip(archive):
+    with ZipFile(archive) as zip:
+        return set(itertools.chain.from_iterable(
+            Path(file).parents for file in zip.namelist()))
+
+
+class PluginFinder(importlib.abc.MetaPathFinder):
+    """
+    This class provides one or multiple namespace packages.
+    It searches in sys.path and yt-dlp config folders for
+    the existing subdirectories from which the modules can be imported
+    """
+
+    def __init__(self, *packages):
+        self._zip_content_cache = {}
+        self.packages = set(itertools.chain.from_iterable(
+            itertools.accumulate(name.split('.'), lambda a, b: '.'.join((a, b)))
+            for name in packages))
+
+    def search_locations(self, fullname):
+        candidate_locations = []
+
+        def _get_package_paths(*root_paths, containing_folder='plugins'):
+            for config_dir in map(Path, root_paths):
+                plugin_dir = config_dir / containing_folder
+                if not plugin_dir.is_dir():
+                    continue
+                yield from plugin_dir.iterdir()
+
+        # Load from yt-dlp config folders
+        candidate_locations.extend(_get_package_paths(
+            *get_user_config_dirs('yt-dlp'), *get_system_config_dirs('yt-dlp'),
+            containing_folder='plugins'))
+
+        # Load from yt-dlp-plugins folders
+        candidate_locations.extend(_get_package_paths(
+            get_executable_path(),
+            compat_expanduser('~'),
+            '/etc',
+            os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config'),
+            containing_folder='yt-dlp-plugins'))
+
+        candidate_locations.extend(map(Path, sys.path))  # PYTHONPATH
+
+        parts = Path(*fullname.split('.'))
+        locations = set()
+        for path in dict.fromkeys(candidate_locations):
+            candidate = path / parts
+            if candidate.is_dir():
+                locations.add(str(candidate))
+            elif path.name and any(path.with_suffix(suffix).is_file() for suffix in {'.zip', '.egg', '.whl'}):
+                with contextlib.suppress(FileNotFoundError):
+                    if parts in dirs_in_zip(path):
+                        locations.add(str(candidate))
+        return locations
+
+    def find_spec(self, fullname, path=None, target=None):
+        if fullname not in self.packages:
+            return None
+
+        search_locations = self.search_locations(fullname)
+        if not search_locations:
+            return None
+
+        spec = importlib.machinery.ModuleSpec(fullname, PluginLoader(), is_package=True)
+        spec.submodule_search_locations = search_locations
+        return spec
+
+    def invalidate_caches(self):
+        dirs_in_zip.cache_clear()
+        for package in self.packages:
+            if package in sys.modules:
+                del sys.modules[package]
+
+
+def directories():
+    spec = importlib.util.find_spec(PACKAGE_NAME)
+    return spec.submodule_search_locations if spec else []
+
+
+def iter_modules(subpackage):
+    fullname = f'{PACKAGE_NAME}.{subpackage}'
+    with contextlib.suppress(ModuleNotFoundError):
+        pkg = importlib.import_module(fullname)
+        yield from pkgutil.iter_modules(path=pkg.__path__, prefix=f'{fullname}.')
+
+
+def load_module(module, module_name, suffix):
+    return inspect.getmembers(module, lambda obj: (
+        inspect.isclass(obj)
+        and obj.__name__.endswith(suffix)
+        and obj.__module__.startswith(module_name)
+        and not obj.__name__.startswith('_')
+        and obj.__name__ in getattr(module, '__all__', [obj.__name__])))
+
+
+def load_plugins(name, suffix):
+    classes = {}
+
+    for finder, module_name, _ in iter_modules(name):
+        if any(x.startswith('_') for x in module_name.split('.')):
+            continue
+        try:
+            if sys.version_info < (3, 10) and isinstance(finder, zipimport.zipimporter):
+                # zipimporter.load_module() is deprecated in 3.10 and removed in 3.12
+                # The exec_module branch below is the replacement for >= 3.10
+                # See: https://docs.python.org/3/library/zipimport.html#zipimport.zipimporter.exec_module
+                module = finder.load_module(module_name)
+            else:
+                spec = finder.find_spec(module_name)
+                module = importlib.util.module_from_spec(spec)
+                sys.modules[module_name] = module
+                spec.loader.exec_module(module)
+        except Exception:
+            write_string(f'Error while importing module {module_name!r}\n{traceback.format_exc(limit=-1)}')
+            continue
+        classes.update(load_module(module, module_name, suffix))
+
+    # Compat: old plugin system using __init__.py
+    # Note: plugins imported this way do not show up in directories()
+    # nor are considered part of the yt_dlp_plugins namespace package
+    with contextlib.suppress(FileNotFoundError):
+        spec = importlib.util.spec_from_file_location(
+            name, Path(get_executable_path(), COMPAT_PACKAGE_NAME, name, '__init__.py'))
+        plugins = importlib.util.module_from_spec(spec)
+        sys.modules[spec.name] = plugins
+        spec.loader.exec_module(plugins)
+        classes.update(load_module(plugins, spec.name, suffix))
+
+    return classes
+
+
+sys.meta_path.insert(0, PluginFinder(f'{PACKAGE_NAME}.extractor', f'{PACKAGE_NAME}.postprocessor'))
+
+__all__ = ['directories', 'load_plugins', 'PACKAGE_NAME', 'COMPAT_PACKAGE_NAME']

yt_dlp/postprocessor/__init__.py

@@ -33,14 +33,15 @@ from .movefilesafterdownload import MoveFilesAfterDownloadPP
 from .sponskrub import SponSkrubPP
 from .sponsorblock import SponsorBlockPP
 from .xattrpp import XAttrMetadataPP
-from ..utils import load_plugins
+from ..plugins import load_plugins
 
-_PLUGIN_CLASSES = load_plugins('postprocessor', 'PP', globals())
+_PLUGIN_CLASSES = load_plugins('postprocessor', 'PP')
 
 
 def get_postprocessor(key):
     return globals()[key + 'PP']
 
 
+globals().update(_PLUGIN_CLASSES)
 __all__ = [name for name in globals().keys() if name.endswith('PP')]
 __all__.extend(('PostProcessor', 'FFmpegPostProcessor'))

yt_dlp/utils.py

@@ -18,7 +18,6 @@ import html.entities
 import html.parser
 import http.client
 import http.cookiejar
-import importlib.util
 import inspect
 import io
 import itertools
@@ -5372,22 +5371,37 @@ def get_executable_path():
     return os.path.dirname(os.path.abspath(_get_variant_and_executable_path()[1]))
 
 
-def load_plugins(name, suffix, namespace):
-    classes = {}
-    with contextlib.suppress(FileNotFoundError):
-        plugins_spec = importlib.util.spec_from_file_location(
-            name, os.path.join(get_executable_path(), 'ytdlp_plugins', name, '__init__.py'))
-        plugins = importlib.util.module_from_spec(plugins_spec)
-        sys.modules[plugins_spec.name] = plugins
-        plugins_spec.loader.exec_module(plugins)
-        for name in dir(plugins):
-            if name in namespace:
-                continue
-            if not name.endswith(suffix):
-                continue
-            klass = getattr(plugins, name)
-            classes[name] = namespace[name] = klass
-    return classes
+def get_user_config_dirs(package_name):
+    locations = set()
+
+    # .config (e.g. ~/.config/package_name)
+    xdg_config_home = os.getenv('XDG_CONFIG_HOME') or compat_expanduser('~/.config')
+    config_dir = os.path.join(xdg_config_home, package_name)
+    if os.path.isdir(config_dir):
+        locations.add(config_dir)
+
+    # appdata (%APPDATA%/package_name)
+    appdata_dir = os.getenv('appdata')
+    if appdata_dir:
+        config_dir = os.path.join(appdata_dir, package_name)
+        if os.path.isdir(config_dir):
+            locations.add(config_dir)
+
+    # home (~/.package_name)
+    user_config_directory = os.path.join(compat_expanduser('~'), '.%s' % package_name)
+    if os.path.isdir(user_config_directory):
+        locations.add(user_config_directory)
+
+    return locations
+
+
+def get_system_config_dirs(package_name):
+    locations = set()
+    # /etc/package_name
+    system_config_directory = os.path.join('/etc', package_name)
+    if os.path.isdir(system_config_directory):
+        locations.add(system_config_directory)
+    return locations
 
 
 def traverse_obj(
@@ -6367,3 +6381,10 @@ class FormatSorter:
 # Deprecated
 has_certifi = bool(certifi)
 has_websockets = bool(websockets)
+
+
+def load_plugins(name, suffix, namespace):
+    from .plugins import load_plugins
+    ret = load_plugins(name, suffix)
+    namespace.update(ret)
+    return ret

ytdlp_plugins/extractor/__init__.py

@@ -1,4 +0,0 @@
-# flake8: noqa: F401
-
-# ℹ️ The imported name must end in "IE"
-from .sample import SamplePluginIE

ytdlp_plugins/extractor/sample.py

@@ -1,14 +0,0 @@
-# ⚠ Don't use relative imports
-from yt_dlp.extractor.common import InfoExtractor
-
-
-# ℹ️ Instructions on making extractors can be found at:
-# 🔗 https://github.com/yt-dlp/yt-dlp/blob/master/CONTRIBUTING.md#adding-support-for-a-new-site
-
-class SamplePluginIE(InfoExtractor):
-    _WORKING = False
-    IE_DESC = False
-    _VALID_URL = r'^sampleplugin:'
-
-    def _real_extract(self, url):
-        self.to_screen('URL "%s" successfully captured' % url)

ytdlp_plugins/postprocessor/__init__.py

@@ -1,4 +0,0 @@
-# flake8: noqa: F401
-
-# ℹ️ The imported name must end in "PP" and is the name to be used in --use-postprocessor
-from .sample import SamplePluginPP

ytdlp_plugins/postprocessor/sample.py

@@ -1,26 +0,0 @@
-# ⚠ Don't use relative imports
-from yt_dlp.postprocessor.common import PostProcessor
-
-
-# ℹ️ See the docstring of yt_dlp.postprocessor.common.PostProcessor
-class SamplePluginPP(PostProcessor):
-    def __init__(self, downloader=None, **kwargs):
-        # ⚠ Only kwargs can be passed from the CLI, and all argument values will be string
-        # Also, "downloader", "when" and "key" are reserved names
-        super().__init__(downloader)
-        self._kwargs = kwargs
-
-    # ℹ️ See docstring of yt_dlp.postprocessor.common.PostProcessor.run
-    def run(self, info):
-        if info.get('_type', 'video') != 'video':  # PP was called for playlist
-            self.to_screen(f'Post-processing playlist {info.get("id")!r} with {self._kwargs}')
-        elif info.get('filepath'):  # PP was called after download (default)
-            filepath = info.get('filepath')
-            self.to_screen(f'Post-processed {filepath!r} with {self._kwargs}')
-        elif info.get('requested_downloads'):  # PP was called after_video
-            filepaths = [f.get('filepath') for f in info.get('requested_downloads')]
-            self.to_screen(f'Post-processed {filepaths!r} with {self._kwargs}')
-        else:  # PP was called before actual download
-            filepath = info.get('_filename')
-            self.to_screen(f'Pre-processed {filepath!r} with {self._kwargs}')
-        return [], info  # return list_of_files_to_delete, info_dict