ArchiveBox
_{Open-source self-hosted web archiving.}

# Option A: Get ArchiveBox with Docker Compose (recommended):
mkdir -p ~/archivebox/data && cd ~/archivebox
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml   # edit options in this file as-needed
docker compose run archivebox init --setup
# docker compose run archivebox add 'https://example.com'
# docker compose run archivebox help
# docker compose up




# Option B: Or use it as a plain Docker container:
mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -it -v $PWD:/data archivebox/archivebox init --setup
# docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'
# docker run -it -v $PWD:/data archivebox/archivebox help
# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox




# Option C: Or install it with your preferred pkg manager (see Quickstart below for apt, brew, and more)
pip install archivebox
mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init --setup
# archivebox add 'https://example.com'
# archivebox help
# archivebox server 0.0.0.0:8000




# Option D: Or use the optional auto setup script to install it
curl -fsSL 'https://get.archivebox.io' | sh

docker-compose (macOS/Linux/Windows)   👈  recommended   (click to expand)

1. Install Docker on your system (if not already installed).
2. Download the docker-compose.yml file into a new empty directory (can be anywhere).

   mkdir -p ~/archivebox/data && cd ~/archivebox
   # Read and edit docker-compose.yml options as-needed after downloading
   curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
   

3. Run the initial setup to create an admin user (or set ADMIN_USER/PASS in docker-compose.yml)

   docker compose run archivebox init --setup
   

4. Next steps: Start the server then login to the Web UI http://127.0.0.1:8000 ⇢ Admin.

   docker compose up
   # completely optional, CLI can always be used without running a server
   # docker compose run [-T] archivebox [subcommand] [--help]
   docker compose run archivebox add 'https://example.com'
   docker compose run archivebox help
   

   For more info, see Install: Docker Compose in the Wiki. ➡️

docker run (macOS/Linux/Windows)

1. Install Docker on your system (if not already installed).
2. Create a new empty directory and initialize your collection (can be anywhere).

   mkdir -p ~/archivebox/data && cd ~/archivebox/data
   docker run -v $PWD:/data -it archivebox/archivebox init --setup
   

3. Optional: Start the server then login to the Web UI http://127.0.0.1:8000 ⇢ Admin.

   docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox
   # completely optional, CLI can always be used without running a server
   # docker run -v $PWD:/data -it [subcommand] [--help]
   docker run -v $PWD:/data -it archivebox/archivebox help
   

   For more info, see Install: Docker Compose in the Wiki. ➡️

bash auto-setup script (macOS/Linux)

1. Install Docker on your system (optional, highly recommended but not required).
2. Run the automatic setup script.

   curl -fsSL 'https://get.archivebox.io' | sh

   For more info, see Install: Bare Metal in the Wiki. ➡️

pip (macOS/Linux/BSD)

1. Install Python >= v3.10 and Node >= v18 on your system (if not already installed).
2. Install the ArchiveBox package using pip3 (or pipx).

   pip3 install --upgrade archivebox yt-dlp playwright
   playwright install --with-deps chromium
   archivebox version
   # install any missing extras shown using apt/brew/pkg/etc. see Wiki for instructions
   #    python@3.10 node curl wget git ripgrep ...
   

   See the Install: Bare Metal Wiki for full install instructions for each OS...
3. Create a new empty directory and initialize your collection (can be anywhere).

   mkdir -p ~/archivebox/data && cd ~/archivebox/data   # for example
   archivebox init --setup   # instantialize a new collection
   # (--setup auto-installs and link JS dependencies: singlefile, readability, mercury, etc.)
   

4. Optional: Start the server then login to the Web UI http://127.0.0.1:8000 ⇢ Admin.

   archivebox server 0.0.0.0:8000
   # completely optional, CLI can always be used without running a server
   # archivebox [subcommand] [--help]
   archivebox help
   

brew (macOS only)

1. Install Homebrew on your system (if not already installed).
2. Install the ArchiveBox package using brew.

   brew tap archivebox/archivebox
   brew install archivebox
   # update to newest version with pip (sometimes brew package is outdated)
   pip install --upgrade --ignore-installed archivebox yt-dlp playwright
   playwright install --with-deps chromium    # install chromium and its system dependencies
   archivebox version                         # make sure all dependencies are installed
   

   See the Install: Bare Metal Wiki for more granular instructions for macOS... ➡️
3. Create a new empty directory and initialize your collection (can be anywhere).

   mkdir -p ~/archivebox/data && cd ~/archivebox/data
   archivebox init --setup
   

4. Optional: Start the server then login to the Web UI http://127.0.0.1:8000 ⇢ Admin.

   archivebox server 0.0.0.0:8000
   # completely optional, CLI can always be used without running a server
   # archivebox [subcommand] [--help]
   archivebox help
   

   
   

pacman / pkg / nix (Arch/FreeBSD/NixOS/more)

• Arch: yay -S archivebox (contributed by @imlonghao)
• FreeBSD: curl -fsSL 'https://get.archivebox.io' | sh (uses pkg + pip3 under-the-hood)
• Nix: nix-env --install archivebox (contributed by @siraben)
• Guix: guix install archivebox (contributed by @rakino)
• More: contribute another distribution...!

docker + electron Desktop App (macOS/Linux/Windows)

1. Install Docker on your system (if not already installed).
2. Download a binary release for your OS or build the native app from source
   
   • macOS: ArchiveBox.app.zip
   • Linux: ArchiveBox.deb (alpha: build manually)
   • Windows: ArchiveBox.exe (beta: build manually)

TrueNAS / UNRAID / YunoHost / Cloudron / etc. (self-hosting solutions)

• TrueNAS: Official ArchiveBox TrueChart / Custom App Guide (TrueCharts is discontinued, wait for Electric Eel)
• UnRaid
• Yunohost
• Cloudron
• Saltbox
• Portainer
• AppImage
• Runtipi
• Umbrel (need contributors...)
• More: contribute another distribution...!

Paid hosting solutions (cloud VPS)

• (get hosting, support, and feature customization directy from us)
• (generalist consultancy that has ArchiveBox experience)

Other providers of paid ArchiveBox hosting (not officially endorsed):



• 
• (USD $29-250/mo, pricing)
• (from USD $2.6/mo)
• (USD $5-50+/mo, 🎗  referral link, instructions)
• (USD $2.5-50+/mo, 🎗  referral link, instructions)
• (USD $10-50+/mo, instructions)
• (USD $0-5+/mo)
• (USD $60-200+/mo)
• (USD $60-200+/mo)

_{Referral links marked 🎗 provide $5-10 of free credit for new users and help pay for our demo server hosting costs.}

CLI Usage Examples: non-Docker

# make sure you have pip-installed ArchiveBox and it's available in your $PATH first  


# archivebox [subcommand] [--help]
archivebox init --setup      # safe to run init multiple times (also how you update versions)
archivebox version           # get archivebox version info + check dependencies
archivebox help              # get list of archivebox subcommands that can be run
archivebox add --depth=1 'https://news.ycombinator.com'

CLI Usage Examples: Docker Compose

# make sure you have `docker-compose.yml` from the Quickstart instructions first


# docker compose run archivebox [subcommand] [--help]
docker compose run archivebox init --setup
docker compose run archivebox version
docker compose run archivebox help
docker compose run archivebox add --depth=1 'https://news.ycombinator.com'
# to start webserver: docker compose up

CLI Usage Examples: Docker

# make sure you create and cd into in a new empty directory first  


# docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
docker run -v $PWD:/data -it archivebox/archivebox init --setup
docker run -v $PWD:/data -it archivebox/archivebox version
docker run -v $PWD:/data -it archivebox/archivebox help
docker run -v $PWD:/data -it archivebox/archivebox add --depth=1 'https://news.ycombinator.com'
# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox

archivebox shell           # explore the Python library API in a REPL
sqlite3 ./index.sqlite3    # run SQL queries directly on your index
ls ./archive/*/index.html  # or inspect snapshot data directly on the filesystem

# Start the server on bare metal (pip/apt/brew/etc):
archivebox manage createsuperuser              # create a new admin user via CLI
archivebox server 0.0.0.0:8000                 # start the server


# Or with Docker Compose:
nano docker-compose.yml                        # setup initial ADMIN_USERNAME & ADMIN_PASSWORD
docker compose up                              # start the server


# Or with a Docker container:
docker run -v $PWD:/data -it archivebox/archivebox archivebox manage createsuperuser
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox

archivebox config --set PUBLIC_ADD_VIEW=True   # allow guests to submit URLs 
archivebox config --set PUBLIC_SNAPSHOTS=True  # allow guests to see snapshot content
archivebox config --set PUBLIC_INDEX=True      # allow guests to see list of all snapshots
# or
docker compose run archivebox config --set ...

# restart the server to apply any config changes

archivebox add --depth=1 'https://example.com'                     # add a URL with pip-installed archivebox on the host
docker compose run archivebox add --depth=1 'https://example.com'                       # or w/ Docker Compose
docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com'  # or w/ Docker, all equivalent

• Index: index.html & index.json HTML and JSON index files containing metadata and details
• Title, Favicon, Headers Response headers, site favicon, and parsed site title
• SingleFile: singlefile.html HTML snapshot rendered with headless Chrome using SingleFile
• Wget Clone: example.com/page-name.html wget clone of the site with warc/TIMESTAMP.gz
• Chrome Headless
  • PDF: output.pdf Printed PDF of site using headless chrome
  • Screenshot: screenshot.png 1440x900 screenshot of site using headless chrome
  • DOM Dump: output.html DOM Dump of the HTML after rendering using headless chrome
• Article Text: article.html/json Article text extraction using Readability & Mercury
• Archive.org Permalink: archive.org.txt A link to the saved site on archive.org
• Audio & Video: media/ all audio/video files + playlists, including subtitles & metadata w/ yt-dlp
• Source Code: git/ clone of any repository found on GitHub, Bitbucket, or GitLab links
• More coming soon! See the Roadmap...

archivebox config                               # view the entire config
archivebox config --get CHROME_BINARY           # view a specific value


archivebox config --set CHROME_BINARY=chromium  # persist a config using CLI
# OR
echo CHROME_BINARY=chromium >> ArchiveBox.conf  # persist a config using file
# OR
env CHROME_BINARY=chromium archivebox ...       # run with a one-off config

# e.g. archivebox config --set TIMEOUT=120
# or   docker compose run archivebox config --set TIMEOUT=120


TIMEOUT=240                # default: 60    add more seconds on slower networks
CHECK_SSL_VALIDITY=False   # default: True  False = allow saving URLs w/ bad SSL
SAVE_ARCHIVE_DOT_ORG=False # default: True  False = disable Archive.org saving
MAX_MEDIA_SIZE=1500m       # default: 750m  raise/lower youtubedl output size


PUBLIC_INDEX=True          # default: True  whether anon users can view index
PUBLIC_SNAPSHOTS=True      # default: True  whether anon users can view pages
PUBLIC_ADD_VIEW=False      # default: False whether anon users can add new URLs


CHROME_USER_AGENT="Mozilla/5.0 ..."  # change these to get around bot blocking
WGET_USER_AGENT="Mozilla/5.0 ..."
CURL_USER_AGENT="Mozilla/5.0 ..."

TIP: For better security while running ArchiveBox, and to avoid polluting your host system with a bunch of sub-dependencies that you need to keep up-to-date,it is strongly recommended to use the ⭐️ official Docker image which provides everything in an easy container with simple one-liner upgrades.

• Language: Python >=3.10
• Backend: Django + Django-Ninja for REST API
• Frontend: Django Admin + Vanilla HTML, CSS, JS
• Web Server: Django + channels + daphne]
• Database: Django ORM saving to SQLite3 ./data/index.sqlite
• Job Queue: Huey using ./data/queue.sqlite3 under supervisord
• Build/test/lint: pdm / mypy+pyright+pytest / ruff
• Subdependencies: pydantic-pkgr installs apt/brew/pip/npm pkgs at runtime (e.g. yt-dlp, singlefile, readability, git)

• chromium / chrome (for screenshots, PDF, DOM HTML, and headless JS scripts)
• node & npm (for readability, mercury, and singlefile)
• wget (for plain HTML, static files, and WARC saving)
• curl (for fetching headers, favicon, and posting to Archive.org)
• yt-dlp or youtube-dl (for audio, video, and subtitles)
• git (for cloning git repos)
• singlefile (for saving into a self-contained html file)
• postlight/parser (for discussion threads, forums, and articles)
• readability (for articles and long text content)
• and more as we grow...

#install python3 and archivebox with your system package manager
# apt/brew/pip/etc install ... (see Quickstart instructions above)


which -a archivebox    # see where you have installed archivebox
archivebox setup       # auto install all the extractors and extras
archivebox --version   # see info and check validity of installed dependencies

Learn More

• Wiki: Install (Dependencies)
• Wiki: Chromium Install
• Wiki: Upgrading or Merging Archives
• Wiki: Troubleshooting (Installing)

mkdir -p ~/archivebox/data && cd ~/archivebox/data   # just an example, can be anywhere
archivebox init

data/
    index.sqlite3
    ArchiveBox.conf
    archive/
        ...
        1617687755/
            index.html
            index.json
            screenshot.png
            media/some_video.mp4
            warc/1617687755.warc.gz
            git/somerepo.git
            ...

Learn More

• Wiki: Setting Up Storage (SMB, NFS, S3, B2, Google Drive, etc.)
• Wiki: Usage (Disk Layout)
• Wiki: Usage (Large Archives)
• Wiki: Security Overview (Output Folder)
• Wiki: Publishing Your Archive
• Wiki: Upgrading or Merging Archives

NOTE: These exports are not paginated, exporting many URLs or the entire archive at once may be slow. Use the filtering CLI flags on the archivebox list command to export specific Snapshots or ranges.

# do a one-off single URL archive wihout needing a data dir initialized
archivebox oneshot 'https://example.com'

# archivebox list --help
archivebox list --html --with-headers > index.html     # export to static html table
archivebox list --json --with-headers > index.json     # export to json blob
archivebox list --csv=timestamp,url,title > index.csv  # export to csv spreadsheet

# (if using Docker Compose, add the -T flag when piping)
# docker compose run -T archivebox list --html 'https://example.com' > index.json

Learn More

• Wiki: Publishing Your Archive (Exporting as Static HTML)
• Wiki: Security Overview (Publishing)
• Wiki: Configuration (PUBLIC_INDEX, PUBLIC_SNAPSHOTS, PUBLIC_ADD_VIEW)

# don't save private content to ArchiveBox, e.g.:
archivebox add 'https://docs.google.com/document/d/12345somePrivateDocument'
archivebox add 'https://vimeo.com/somePrivateVideo'

# without first disabling saving to Archive.org:
archivebox config --set SAVE_ARCHIVE_DOT_ORG=False  # disable saving all URLs in Archive.org

# restrict the main index, Snapshot content, and Add Page to authenticated users as-needed:
archivebox config --set PUBLIC_INDEX=False
archivebox config --set PUBLIC_SNAPSHOTS=False
archivebox config --set PUBLIC_ADD_VIEW=False 
archivebox manage createsuperuser

# if extra paranoid or anti-Google:
archivebox config --set SAVE_FAVICON=False          # disable favicon fetching (it calls a Google API passing the URL's domain part only)
archivebox config --set CHROME_BINARY=chromium      # ensure it's using Chromium instead of Chrome

CAUTION: Assume anyone viewing your archives will be able to see any cookies, session tokens, or private URLs passed to ArchiveBox during archiving. Make sure to secure your ArchiveBox data and don't share snapshots with others without stripping out sensitive headers and content first.

Learn More

• Wiki: Publishing Your Archive
• Wiki: Security Overview
• Wiki: Chromium Install (Setting Up a User Profile)
• Wiki: Configuration (CHROME_USER_DATA_DIR)
• Wiki: Configuration (COOKIES_FILE)

# visiting an archived page with malicious JS:
https://127.0.0.1:8000/archive/1602401954/example.com/index.html

# example.com/index.js can now make a request to read everything from:
https://127.0.0.1:8000/index.html
https://127.0.0.1:8000/archive/*
# then example.com/index.js can send it off to some evil server

NOTE: Only the wget & dom extractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing.
If you are worried about these issues ^ you should disable these extractors using:
archivebox config --set SAVE_WGET=False SAVE_DOM=False.

Learn More

• Wiki: Security Overview
• ArchiveBox Github Issue: #239
• Security Advisory: CVE-2023-45815
• Wiki: Security Overview (Publishing)

• Set CHROME_USER_AGENT, WGET_USER_AGENT, CURL_USER_AGENT to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)
• Set up a logged-in browser session for archiving using CHROME_USER_DATA_DIR & COOKIES_FILE
• Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.
  reddit.com/some/url -> teddit.net/some/url: https://github.com/mendel5/alternative-front-ends

archivebox add 'https://example.com#2020-10-24'
...
archivebox add 'https://example.com#2020-10-25'

Learn More

• ArchiveBox Issues: #179
• Wiki: Usage (Explanation of Web UI Buttons)

• ArchiveBox can use anywhere from ~1gb per 1000 Snapshots, to ~50gb per 1000 Snapshots, mostly dependent on whether you're saving audio & video using SAVE_MEDIA=True and whether you lower MEDIA_MAX_SIZE=750mb.
• Disk usage can be reduced by using a compressed/deduplicated filesystem like ZFS/BTRFS, or by turning off extractors methods you don't need. You can also deduplicate content with a tool like fdupes or rdfind.
• Don't store large collections on older filesystems like EXT3/FAT as they may not be able to handle more than 50k directory entries in the data/archive/ folder.
• Try to keep the data/index.sqlite3 file on local drive (not a network mount) or SSD for maximum performance, however the data/archive/ folder can be on a network mount or slower HDD.
• If using Docker or NFS/SMB/FUSE for the data/archive/ folder, you may need to set PUID & PGID and disable root_squash on your fileshare server.

Learn More

• Wiki: Usage (Disk Layout)
• Wiki: Security Overview (Output-Folder)
• Wiki: Usage (Large Archives)
• Wiki: Configuration (PUID & GUID)
• Wiki: Security Overview (Do Not Run as Root)

Comparison With Centralized Public Archives

Comparison With Other Self-Hosted Archiving Options