Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-25 21:50:21 +00:00
Code Issues Projects Releases Packages Wiki Activity
4117 commits 6 branches 78 tags 19 MiB
Commit graph

262 commits

Author SHA1 Message Date
mAAdhaTTah
181501fd36
Add Readwise Reader API parser 
Implemented similar to the Pocket API.
 2023-07-02 11:20:58 -04:00
Micah R Ledbetter
1e50ca243e Add FAVICON_PROVIDER option for custom favicon service 2023-05-05 20:42:36 -05:00
Nick Sweeting
9f42a3bf29
fix whitespace 2023-03-15 16:01:02 -07:00
ふぁ
d77c770c47
add CHROME_TIMEOUT args 
Signed-off-by: ふぁ <yuki@yuki0311.com>
 2023-03-14 20:29:41 +09:00
Nick Sweeting
dca69933eb
Update archivebox/config.py 
Co-authored-by: dugite-code <dugite-code@users.noreply.github.com>
 2023-01-09 18:22:01 -08:00
Nick Sweeting
2538b170c7
Merge branch 'dev' into feat/reverse-proxy-auth 2023-01-09 18:20:45 -08:00
Nick Sweeting
0cbeeb4346
Merge pull request #1021 from renaisun/dev 2023-01-09 18:17:39 -08:00
Joseph Turian
30947aeb07 yt-dlp flag cleanup 2022-09-14 06:29:57 +02:00
Joseph Turian
f729bbe122 yt-dlp fixes 2022-09-14 06:27:58 +02:00
Joseph Turian
dba423a568 A few more youtube-dl tweaks 2022-09-12 20:36:23 +00:00
Joseph Turian
f5f7aff3b4 Added yt-dlp everywhere 2022-09-12 20:34:02 +00:00
Nick Sweeting
03eb7e5875
Update config.py 2022-06-09 01:04:55 -07:00
renaisun
8899fe0b92
Add SINGLEFILE_ARGS to control single-file arguments 2022-06-09 14:35:48 +08:00
Nick Sweeting
319ea481b8
Update config.py 2022-06-08 20:17:38 -07:00
Nick Sweeting
2bbc742017
typo fix 2022-06-08 20:16:08 -07:00
Nick Sweeting
e2fa68dba6
resolve config paths before using 2022-06-08 20:15:22 -07:00
Nick Sweeting
9f90a2d60d
disable unused sqlite3 stuff 2022-06-08 20:12:55 -07:00
Nick Sweeting
c78a2edc42
add is_mount and COMMIT_HASH to config.py 2022-06-08 20:04:01 -07:00
Nick Sweeting
0c6d4c82c3
Update config.py 2022-06-08 19:11:02 -07:00
Nick Sweeting
f9c5808940
Update config.py 2022-06-08 19:09:11 -07:00
Nick Sweeting
31d5fbbf17
Update config.py 2022-06-08 19:04:06 -07:00
Nick Sweeting
6b019da3e9
Update config.py 2022-06-08 19:01:55 -07:00
Nick Sweeting
f9c82841ad
fix sqlite option detection 2022-06-08 18:58:15 -07:00
Nick Sweeting
3e3c011f86
enforce UTC timezone on server 2022-06-08 18:40:48 -07:00
Nick Sweeting
e06717419c
fix sqlite3 version detection 2022-06-08 18:35:31 -07:00
Nick Sweeting
d0f129295f
move sqlite3 checks up a level 2022-06-08 18:29:53 -07:00
Nick Sweeting
89175ccb22
check SQLite3 version and enabled extensions on startup 2022-06-08 18:24:17 -07:00
Nick Sweeting
c245d36e44
add PUID and PGID as config options in archivebox 2022-06-08 17:42:52 -07:00
Nick Sweeting
5e4ddbbf25 fix mercury bin parsing back 2022-05-09 21:58:17 -07:00
Nick Sweeting
e96c1bcf13 bump mercury parser to git head version 2022-05-09 21:48:41 -07:00
Nick Sweeting
d05510f844 fix version string parsing on macOS in some cases where LANG C is not supported 2022-05-09 21:21:08 -07:00
Nick Sweeting
8ebf3e2f93 add config option PREVIEW_ORIGINALS to hide original iframes in snapshot detail pages 2022-05-09 19:31:41 -07:00
Nick Sweeting
6e66863871 add max 5s writing delay for concurrent writers and flush WAL slower 2022-05-09 18:36:40 -07:00
Nick Sweeting
57df65f28f use yt-dlp for media archiving instead of youtube-dl 2022-04-21 07:11:35 -07:00
Nick Sweeting
eb81d41f84 bump Dockerfile base image version and install yt-dlp 2022-04-21 07:11:35 -07:00
Igor Rzegocki
d4f534e612
add LOGOUT_REDIRECT_URL 2022-03-31 21:40:14 +02:00
Pellaeon Lin
5e9d05483e Fix bin_version: set LANG=C when calling executables to avoid parsing 
localized output.
 2022-02-24 17:01:00 +08:00
Nick Sweeting
666ab20df5
Update archivebox/config.py 2022-01-10 20:42:09 -05:00
hannah98
fc3d2bb4dc rename TAG_SEPARATORS to TAG_SEPARATOR_PATTERN 2022-01-06 14:14:41 +00:00
hannah98
049f88def9 Added TAG_SEPARATORS option to supply a regex of characters to use when splitting tags 2021-12-30 20:19:48 +00:00
Mika Tuupola
f14a861605 Change logfile open to write mode only 2021-12-19 23:17:33 +02:00
Igor Rzegocki
95cf85f8cf
Support for Reverse Proxy authentication backends (like authelia) 2021-09-30 17:40:13 +02:00
Nick Sweeting
5a2c78e14b add proper support for URL_WHITELIST instead of using negation regexes 2021-07-06 23:42:00 -04:00
Nick Sweeting
c2d1a57581 fix umask dir permissions 2021-06-01 00:50:18 -04:00
Nick Sweeting
8230f88d80 change default OUTPUT_PERMISSIONS to disallow execution except on dirs 2021-05-31 19:31:51 -04:00
Nick Sweeting
1112526543 add option ENFORCE_ATOMIC_WRITES to allow disabling forced FSYNC writes on network drives 2021-05-31 19:31:51 -04:00
Nick Sweeting
79e19ecd47 shield pwd import for windows 2021-04-24 03:51:38 -04:00
Nick Sweeting
208f866fc4 fix user detection on freebsd always returning root 2021-04-24 02:52:09 -04:00
Nick Sweeting
9563e74f7c enforce minimum django version 2021-04-23 15:02:26 -04:00
Nick Sweeting
54cab113f3 allow static_index.json in dir root from old static exports 2021-04-12 17:06:32 -04:00
Nick Sweeting
6a6ef63775 enforce utf8 on more file reads 2021-04-10 07:22:32 -04:00
Nick Sweeting
193df5c8d3 add video subtitles and description to full-text index 2021-04-10 07:22:20 -04:00
Nick Sweeting
1e3c1e3c83 also write autogenerated subtitles 2021-04-10 06:56:19 -04:00
Nick Sweeting
c5a9b77f4f update user agents to mimic latest chrome 2021-04-10 05:45:02 -04:00
Nick Sweeting
a9986f1f05 add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support 2021-04-10 04:21:36 -04:00
Nick Sweeting
06c58ac42a fix unreachable config backup clear 2021-04-09 12:15:47 -04:00
Nick Sweeting
32764347ce add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec 2021-04-09 00:27:12 -04:00
Nick Sweeting
2e57df917e handle BaseExceptions properly 2021-04-08 06:08:17 -04:00
Nick Sweeting
2c07d8f1f3 fix lgtm alerts 2021-04-06 13:47:26 -04:00
Nick Sweeting
e92db03f5f fallback to using bin_path to find archivebox binary if sys.argv is empty 2021-04-06 02:45:57 -04:00
Nick Sweeting
e4823cc5d7 add back CUSTOM_TEMPLATES_DIR config option 2021-04-06 00:57:20 -04:00
Nick Sweeting
5a9f27204a dont use chrome when its not available on windows systems 2021-04-05 23:33:08 -04:00
Nick Sweeting
bbebc33b7a better autodetection of node modules dependencies 2021-04-05 21:14:06 -04:00
Nick Sweeting
ef76f7c490 require running setup in existing data dir 2021-04-05 19:59:33 -04:00
Nick Sweeting
3449bac5bc add new setup command to autoinstall dependencies 2021-04-05 19:51:57 -04:00
Nick Sweeting
0e39a2098d split migrations and data dir check 2021-04-01 14:00:07 -04:00
Nick Sweeting
7e7c134ec0 use exc 2021-03-27 05:32:07 -04:00
Nick Sweeting
bd6d9c165b enforce utf8 on literally all file operations because windows sucks 2021-03-27 01:16:29 -04:00
Nick Sweeting
185d2f9f9b check for JSON1 extesnion in Sqlite on startup 2021-03-27 00:27:58 -04:00
Nick Sweeting
62089fdb22 close sqlite3 connections if unused 2021-02-28 22:53:34 -05:00
Nick Sweeting
e3b69c450c move db checks under check_db if statement 2021-02-18 02:37:33 -05:00
Nick Sweeting
c877d67e0c check if pragma is already wal before setting 2021-02-18 02:31:42 -05:00
Nick Sweeting
7b7aa239fd autocreate db cache table when needed and reenable WAL mode 2021-02-17 18:26:04 -05:00
Nick Sweeting
6f0eec92eb fix lint errors 2021-02-16 16:26:48 -05:00
Nick Sweeting
8cde6bc676 add gist.github.com to git domains 2021-02-16 07:36:09 -05:00
Nick Sweeting
46a7b972ee disable WAL and cache table for now 2021-02-16 06:13:50 -05:00
Nick Sweeting
fb8e6cabcb fix error log location 2021-02-16 04:15:09 -05:00
Nick Sweeting
19f7c907e0 exec archivebox from docker entrypoint script to avoid nesting pid under bash 2021-02-16 01:22:36 -05:00
Nick Sweeting
10d687c55c add new MEDIA_MAX_SIZE=750m size option for limiting youtubedl downloads 2021-02-16 01:21:19 -05:00
Nick Sweeting
0c9db1c554 remove symbols from random secret key for easier copy pastin 2021-02-15 20:45:42 -05:00
Nick Sweeting
78463c243a remove unused GIT_SHA config option 2021-02-15 20:42:33 -05:00
Nick Sweeting
9cd4ba38f0 add new SNAPSHOTS_PER_PAGE pagination limit config 2021-02-15 20:42:00 -05:00
Nick Sweeting
e61e12c889 use setup.py to determine dependencies in Dockerfile instead of egg-info requires.txt 2021-02-15 14:51:32 -05:00
Nick Sweeting
683a08772b change wording of db not found error 2021-02-08 23:27:46 -05:00
Nick Sweeting
7d0f5653c3 fix lgtm alerts 2021-02-01 02:27:24 -05:00
Nick Sweeting
d072f1d413 hide ssl warnings when checking SSL is disabled 2021-01-30 20:44:49 -05:00
Nick Sweeting
9d24bfd0dc disable progress bars on mac again 2021-01-30 20:44:49 -05:00
Nick Sweeting
ed13ec7655 remove active theme 2021-01-30 05:47:55 -05:00
Nick Sweeting
3227f54b52 limit youtubedl download size to 750m and stop splitting out audio files 2021-01-29 00:15:32 -05:00
Dan Arnfield
5c7842ffb3 Fix dependency dict entries 2021-01-20 09:24:34 -06:00
Nick Sweeting
9784dcb816 better config comments and docstrings 2020-12-20 03:11:19 +02:00
jdcaballerov
50df108863
Update archivebox/config.py 
Co-authored-by: Nick Sweeting <git@sweeting.me>
 2020-12-12 08:34:00 -05:00
jdcaballerov
24d4c44624 Add ripgrep configs 2020-12-12 07:36:31 -05:00
Nick Sweeting
335732649b tweak node dependency version detection order 2020-12-11 21:03:17 +02:00
Nick Sweeting
2db5e51b54 fix windows shutil not able to handle pathlib 2020-12-11 19:33:18 +02:00
Nick Sweeting
6623497f18 fix MERCURY_PATH in version output when missing 2020-12-11 16:21:33 +02:00
Nick Sweeting
c084e70ea8 fix TEMPLATES_DIR location 2020-12-11 16:21:09 +02:00
Cristian
9745a5ac56 fix: Migrations should be silent when running in setup_django 2020-12-08 18:42:25 -05:00
Cristian
9aac09a5e1 feat: Patch setup_django so we can use an inmemory db in specific commands 2020-12-08 18:42:25 -05:00
JDC
95382b3812 Add ripgrep rg search backend and set as default 2020-12-06 01:13:39 +02:00
JDC
47daa038eb Implement flush for search backend after remove command 2020-12-06 01:12:47 +02:00
JDC
c2c01af3ad Add config for search backend 2020-12-06 01:12:47 +02:00
mAAdhaTTah
ac7ad9e942
Add parser for Pocket API 
Pass a url like `pocket://Username` to import that username's archived Pocket
library. Tokens need to be stored in ArchveBox.conf with the following keys:

```
POCKET_CONSUMER_KEY = key-from-custom-pocket-app
POCKET_ACCESS_TOKENS = {"YourUsername": "pocket-token-for-app"}
```

`POCKET_ACCESS_TOKENS` MUST be on a single line, or the JSON will be
misinterpreted by the parser as a new key/value pair.
 2020-12-04 22:54:39 -05:00
Nick Sweeting
7f2c834ea3 fix check_data_folder mypy types 2020-11-28 01:05:35 -05:00
Nick Sweeting
5e7c2d0ab8 show archivebox and node versions in version cmd output 2020-11-23 20:24:44 -05:00
Nick Sweeting
0e2ccbc10d update urls to new repo path 2020-11-23 02:06:46 -05:00
Nick Sweeting
3f160eab8e correctly handle WGET_AUTO_COMPRESSION failing when wget is missing 2020-11-12 14:28:43 -05:00
Nick Sweeting
22fb9c2ad7 tweak icons 2020-10-31 19:32:43 -04:00
Nick Sweeting
b8bbb75f9c logarithmic progress bars woohoo 2020-10-31 07:57:11 -04:00
Nick Sweeting
ac9e0e356d config fixes 2020-10-31 07:57:11 -04:00
Nick Sweeting
18355dc2c6 clean up config loading in settings and config file layout 2020-10-31 03:08:03 -04:00
Nick Sweeting
27708152d2 wip initial django setup 2019-04-02 16:36:41 -04:00
Nick Sweeting
f4e018ba0c fix a bunch of mypy errors 2019-03-30 20:49:45 -04:00
Nick Sweeting
8283b353f4 fix wget_auto_compression check prining stderr to console 2019-03-30 17:43:44 -04:00
Nick Sweeting
5d0185b6dd Merge branch 'master' into dev 2019-03-30 15:36:54 -04:00
Nick Sweeting
35c05c321f minor bin version checking changes 2019-03-30 15:03:31 -04:00
Nick Sweeting
066b36b6a9
make URL_BLACKLIST empty by default 2019-03-30 14:56:19 -04:00
Nick Sweeting
a26c2fe467 show full version info using flag 2019-03-27 16:44:00 -04:00
Nick Sweeting
93216a3c3e new version handling and absolute imports 2019-03-27 15:35:13 -04:00
Nick Sweeting
b9839500b2 make archivebox use current directory as OUTPUT_DIR by default 2019-03-27 15:15:51 -04:00
mlazana
a3705e31c6 Merge remote-tracking branch 'upstream/master' 2019-03-27 20:03:07 +02:00
Nick Sweeting
7ae680a461 Merge branch 'master' into dev 2019-03-27 10:36:29 -04:00
Nick Sweeting
6bec1709f6
Update config.py 2019-03-27 04:48:38 -04:00
Nick Sweeting
c9c5b04df0 full type-hinting coverage 2019-03-26 23:25:07 -04:00
Starbeamrainbowlabs
51859731d3
config.py: add WGET_AUTO_COMPRESSION 2019-03-26 23:34:45 +00:00
Nick Sweeting
25a107df43 switch to dataclasses, working Link type hints everywhere 2019-03-26 19:21:34 -04:00
Nick Sweeting
52871f42e7 cleaner config with type hints 2019-03-26 05:31:27 -04:00
Nick Sweeting
4f8c99011a fix terminal resizing making progress bar go crazy 2019-03-26 05:30:23 -04:00
mlazana
4d10568477 exclude links that are in blacklist 2019-03-24 14:40:26 +02:00
mlazana
417ee9e302 add env variable URL_BLACKLIST 2019-03-23 21:27:41 +02:00
Nick Sweeting
f7a0568a6c better chrome options loading 2019-03-22 23:00:53 -04:00
Nick Sweeting
4c499d77b6 move dependency checking into config file 2019-03-22 22:05:45 -04:00
Nick Sweeting
69f837bbf6 simplify chrome_user_data_dir default 2019-03-22 21:37:02 -04:00
Nick Sweeting
8f73fdbe09 fix chrome profile precedence order to be equal 2019-03-22 21:31:55 -04:00
noncetonic
28758cf16c
Adds CHROME_USER_AGENT 2019-03-19 10:15:52 -07:00
Nick Sweeting
1c1bc76ac1 add chrome headless option and improve default data dir finding 2019-03-12 17:50:10 -04:00
Nick Sweeting
8319ccf064
add docs link to config.py 2019-03-12 12:48:46 -04:00
Nick Sweeting
32c39d0fd0 cleaner output dir spec in config 2019-03-08 17:51:49 -05:00
Nick Sweeting
2e10f57f6e fix relative links from index files 2019-03-08 17:46:14 -05:00
Nick Sweeting
a74d8410f4 also check for macOS binary defaults 2019-03-08 16:25:42 -05:00
Nick Sweeting
d689264365 add new config and dependency options 2019-02-21 15:47:15 -05:00
Nick Sweeting
d52c9c5304 allow passing COOKIES_FILE to wget 2019-02-21 12:58:51 -05:00
Nick Sweeting
5a7d00a639 fetch page title during archiving process 2019-02-19 01:44:54 -05:00
Nick Sweeting
9eb79258bb check chrome version on startup if using chrome 2019-01-29 17:08:15 -08:00
Nick Sweeting
e60070dbb2 add youtubedl to help str 2019-01-25 17:38:47 -08:00
Nick Sweeting
20de451515 add linux config example 2019-01-23 01:48:04 -05:00
Nick Sweeting
95301c9306 better default config 2019-01-23 01:42:55 -05:00
Nick Sweeting
e1be96e597 working docker-compose with google chrome 2019-01-23 01:08:23 -05:00
Nick Sweeting
f25be8bc24 add chrome user agent example in config 2019-01-20 14:08:33 -05:00
Nick Sweeting
cc8611de83
Update config.py 2019-01-14 22:46:35 -05:00