Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-29 07:30:23 +00:00
Code Issues Projects Releases Packages Wiki Activity
4452 commits 6 branches 78 tags 19 MiB
Commit graph

257 commits

Author SHA1 Message Date
Nick Sweeting
19b88d30b2
fix missing brace 2022-06-08 19:06:03 -07:00
Nick Sweeting
c752c7053d
Update main.py 2022-06-08 18:59:08 -07:00
Nick Sweeting
1fd5830f58
enforce UTC timezone on server 2022-06-08 18:41:22 -07:00
Nick Sweeting
0c7d7deb32
add missing brace 2022-06-08 18:26:42 -07:00
Nick Sweeting
ca16c88a3d
show PUID, PGID, ENFORCE_ATOMIC_WRITES, and OUTPUT_PERMISSIONS in version output header 2022-06-08 18:24:58 -07:00
Nick Sweeting
f6d6a06c78 always show all totals in log output 2022-05-09 21:21:26 -07:00
Nick Sweeting
4b8b17e788 add update flag support to archivebox schedule 2022-05-09 20:18:43 -07:00
Nick Sweeting
8cfe6f4afb cleanup update flag handling and show better logging to clarify when its working 2022-05-09 20:15:55 -07:00
Nick Sweeting
38e54b93fe allow parsing to continue even when fetching URL contents fails 2022-05-09 19:56:24 -07:00
Ross
c63822a5e5
Fix missing input redirection in a hint text 2022-04-19 22:25:49 +01:00
Nick Sweeting
e0a2b2e252 ominous warnings 2021-06-01 03:03:42 -04:00
Nick Sweeting
4715ace7dd ignore BaseException lgtm errors 2021-05-31 20:59:05 -04:00
Nick Sweeting
3db77fd1a2 fix lint errors 2021-04-24 03:49:01 -04:00
Nick Sweeting
fa84f749ff run archivebox version using python binary 2021-04-24 03:29:22 -04:00
Nick Sweeting
226e26852c dont try to autodetect whether node, chrome, etc are needed during setup 2021-04-24 03:29:22 -04:00
Nick Sweeting
f129b9f443 allow executing main 2021-04-24 03:11:06 -04:00
Nick Sweeting
4f91af0ce9 TODO create admin user from env vars 2021-04-23 15:50:34 -04:00
Nick Sweeting
63552be0e3 whoops 2021-04-13 04:03:47 -04:00
Nick Sweeting
7d1f8cb496 no longer recommend snap chromium, just use playwright 2021-04-13 03:59:17 -04:00
Nick Sweeting
54cab113f3 allow static_index.json in dir root from old static exports 2021-04-12 17:06:32 -04:00
Nick Sweeting
a9986f1f05 add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support 2021-04-10 04:21:36 -04:00
Nick Sweeting
cadac4838e fix searching by url contents 2021-04-05 23:21:07 -04:00
Nick Sweeting
2fc9f1cfa3 dont depend on README in Dockerfile install step 2021-04-05 23:17:07 -04:00
Nick Sweeting
8b4f84959a new setup flag on init command to autosetup on init 2021-04-05 21:15:32 -04:00
Nick Sweeting
5c181532b5 only install dependencies in setup if they are missing 2021-04-05 21:15:10 -04:00
Nick Sweeting
79fb60a47a dont allow caching when installing packages 2021-04-05 20:08:48 -04:00
Nick Sweeting
8c1d12020f fix lint errors 2021-04-05 19:53:42 -04:00
Nick Sweeting
3449bac5bc add new setup command to autoinstall dependencies 2021-04-05 19:51:57 -04:00
Nick Sweeting
24f7dd9c05 dont raise systemexit on succesful config 2021-04-05 19:51:40 -04:00
Nick Sweeting
fc5d99b4b4 fix missing package.json repo 2021-04-01 01:40:02 -04:00
Nick Sweeting
7ee7a94e67 fix str split using old overwrite pipe instead of append pipe 2021-03-31 11:30:15 -04:00
Nick Sweeting
005da31f8f add overwrite flag option to schedule cmd 2021-03-31 11:29:51 -04:00
Nick Sweeting
534ae60ab0 dont overwrite the log on each run, and use dedicated log for scheduled archiving 2021-03-31 11:20:42 -04:00
Nick Sweeting
36f0646501
Merge pull request #669 from FliegendeWurst/fix-issue-235 
add command: --parser option (fixes #235)
 2021-03-31 00:53:47 -04:00
FliegendeWurst
60bd9a902e add command: --parser option 2021-03-28 10:09:11 +02:00
Nick Sweeting
49b77b91bb fix unused var 2021-03-27 05:12:54 -04:00
Nick Sweeting
f3a3d76439 support adding urls with tags directly via CLI and add page 2021-03-27 04:30:15 -04:00
Nick Sweeting
fea0b89dbe add tag cli option 2021-03-27 03:57:05 -04:00
Nick Sweeting
ba2afa0672 allow cancelling init partway through and show hints to resume 2021-02-18 02:34:42 -05:00
Nick Sweeting
bdf1b102be load ArchiveResults from orphaned links history during init 2021-02-16 06:20:05 -05:00
Nick Sweeting
fad2620c62 minor tweaks 2021-02-16 02:49:31 -05:00
Nick Sweeting
4eac9d0ec1 show extra system setup info in version output 2021-02-16 01:23:58 -05:00
Nick Sweeting
b3a50a2c10 fix server quick-init param not being passed properly to subcommand 2021-02-15 20:49:40 -05:00
Nick Sweeting
b06e256ad9 fix add command not updating snapshot detail index pages when passed index-only and overwrite flags together 2021-02-15 20:49:23 -05:00
Nick Sweeting
8e98cef7ad fix after and before args flipped when filtering 2021-02-15 20:48:51 -05:00
Nick Sweeting
00ae1f15a7 ignore shm db file and config files in archivebox data dir on init 2021-02-15 14:52:37 -05:00
Nick Sweeting
3c3bae02d2 add quick-init option to skip reimporting all snapshot dirs on init 2021-02-15 14:52:10 -05:00
Nick Sweeting
611216765d switch sqlite to use WAL mode by default to prevent database locked errors 2021-02-15 13:39:03 -05:00
Nick Sweeting
783f597955 minor build fixes 2021-02-01 05:13:46 -05:00
Nick Sweeting
ed13ec7655 remove active theme 2021-01-30 05:47:55 -05:00
Nick Sweeting
39ec77e46c add createsuperuser flag to server command 2021-01-28 22:27:02 -05:00
Cristian
a28547cbca refactor: Remove get_empty_snapshot queryset function and generate it directly 2020-12-11 16:27:15 -05:00
Nick Sweeting
e90cf05141 fix lint errors 2020-12-11 16:51:11 +02:00
Nick Sweeting
30f8d3f191 show python implementation name and flip verison output order for easier reading when wrapped on small screens 2020-12-11 16:21:52 +02:00
Nick Sweeting
9fa70b3452 add extractors arg to oneshot command and bump version to v0.5.1 2020-12-11 15:48:46 +02:00
Cristian
275ad22db7 refactor: Remove skip_index from archive related functions 2020-12-08 18:42:25 -05:00
Cristian
9aac09a5e1 feat: Patch setup_django so we can use an inmemory db in specific commands 2020-12-08 18:42:25 -05:00
JDC
caf4660ac8 Add indexing to update command and utilities 2020-12-06 01:14:37 +02:00
JDC
c5b1b91708 fix: flush_search_index must be called before removing snapshots 2020-12-06 01:13:39 +02:00
JDC
823df34080 Use QuerySets for search backend API instead of pks 2020-12-06 01:12:47 +02:00
JDC
f383648ffc Use a generator for snapshot flush from index 2020-12-06 01:12:47 +02:00
JDC
47daa038eb Implement flush for search backend after remove command 2020-12-06 01:12:47 +02:00
Nick Sweeting
3b280e6b02
Merge pull request #569 from cdvv7788/extract-command-update 
feat: Add --extract flag to update command
 2020-12-05 17:43:28 -05:00
Cristian
35389608d1 feat: Add --extract flag to update command 2020-12-05 12:20:47 -05:00
Cristian
34cad4fe8d fix: Update function with --index-only flag was not behaving as expected 2020-12-05 12:10:17 -05:00
Cristian
7008f9b735 feat: move import 2020-11-28 13:11:15 -05:00
Cristian
fa5de72f9f refactor: Move indexing logic out of logging module 2020-11-28 12:34:40 -05:00
Nick Sweeting
c9162a6d09 remove finished/not finished spinners 2020-11-28 01:07:02 -05:00
Nick Sweeting
1b22f8eeef
Merge pull request #515 from cdvv7788/POC-setup-django-on-init 2020-11-27 23:56:37 -05:00
Nick Sweeting
e4d2ac432d
improve OS kernel output in archivebox version 2020-11-27 23:08:23 -05:00
Nick Sweeting
07a56f9d46
also print platform and CPU info in version output 2020-11-27 22:59:18 -05:00
Nick Sweeting
d9ef3d0bf8
ignore lost+found dir in data folder 2020-11-27 19:39:19 -05:00
mAAdhaTTah
be7a7f8548
Fix string checks in schedule 
`s` comes through as a `PosixPath`, so both the `' ' in s` & return value, later
used by `join`, complain.
 2020-11-23 18:34:07 -05:00
Nick Sweeting
0e2ccbc10d update urls to new repo path 2020-11-23 02:06:46 -05:00
Nick Sweeting
b11d562445
fix splitting on multiple equals in val 2020-11-22 12:33:15 -05:00
Cristian
54df0a035b fix: Move csv split to the add function to avoid optional nullable argument 2020-11-13 13:10:17 -05:00
Cristian
db523c9d82 fix: Avoid mutable default input argument 2020-11-13 11:41:50 -05:00
Cristian
44eede96e5 feat: Add extract flag to add command 2020-11-13 09:24:34 -05:00
Nick Sweeting
79051ca15b new package build 2020-10-31 03:08:41 -04:00
Nick Sweeting
18355dc2c6 clean up config loading in settings and config file layout 2020-10-31 03:08:03 -04:00
Cristian
e1d0b8bce7 feat: Initialize django at the beginning 2020-10-26 07:45:21 -05:00
Cristian
f397634dd2 feat: Rename old indexes at the end of init process 2020-10-23 06:45:56 -05:00
Cristian
ae1484b8bf feat: Remove index.json and index.html generation from the regular process 2020-10-23 06:45:56 -05:00
Cristian
b18bbf8874 test: Fix tests post-rebase 2020-09-17 09:09:52 -05:00
apkallum
594d9e49ce first attempt to migrate to Pathlib 2020-09-17 09:09:52 -05:00
Cristian
5348f4735a fix: Change check to avoid issues with empty querysets 2020-09-15 08:05:46 -05:00
Cristian
cf18130f85 feat: Add deprecation warning for index.json 2020-09-15 08:05:46 -05:00
Cristian
fa622d3e14 refactor: Replace --index with --with-headers in the list command to make it more explicit. Change it so it affects the csv output too. 2020-09-15 08:05:46 -05:00
Cristian
2aa8d69b72 fix: Save history in main index (to mimic previous behaviour) 2020-09-15 08:05:46 -05:00
Cristian
7e9d195d13 feat: Update list command to sort using sqlite 2020-09-15 08:05:46 -05:00
Cristian
f55153eab3 feat: Update update command to work with querysets 2020-09-15 08:05:46 -05:00
Cristian
a8ed72501d feat: Refactor remove command to use querysets 2020-09-15 08:05:46 -05:00
Cristian
be520d137a feat: Refactor add method to use querysets 2020-09-15 08:05:46 -05:00
Cristian
6a2e6aad2f fix: status command was failing on empty archives 2020-09-15 08:05:46 -05:00
Cristian
be0dff8126 feat: Add tests to refactored init command 2020-09-15 08:05:46 -05:00
Cristian
404f333e17 feat: Refactor get_invalid_folders to work with a queryset instead of a list of links 2020-09-15 08:05:46 -05:00
Cristian
dae606de6e feat: Update init to take advantage of querysets to reduce memory consumption 2020-09-15 08:05:46 -05:00
Cristian
6b4b7127b4 feat: Remove unused imports 2020-09-15 08:05:46 -05:00
Cristian
b8585dd92e feat: load_main_index returns a queryset now 2020-09-15 08:05:46 -05:00
Cristian
885ff50449 feat: Add html export to list command 2020-09-15 08:05:46 -05:00
Cristian
aab8f96520 feat: Add flag to list command to support index like output 2020-09-15 08:05:46 -05:00
Cristian
be57db1369 feat: Save static indexes at the end of init 2020-09-15 08:05:46 -05:00
Cristian
c16fdf1b47 feat: Update data folder check 2020-09-15 08:05:46 -05:00
Cristian
874403e667 feat: Remove patch_main_index 2020-09-15 08:05:46 -05:00
Cristian
31343c1367 feat: Update extractors and add command to use sql index as source of truth 2020-09-15 08:05:46 -05:00
Cristian
e9caee6b10 feat: Update status command to consider sql as the main index 2020-09-15 08:05:46 -05:00
Cristian
02f36b2096 feat: Replace index.json with index.sql as the main index in init 2020-09-15 08:05:46 -05:00
Nick Sweeting
5cb13ff1dd ignore node cruft in output folder during init 2020-08-18 18:20:26 -04:00
Nick Sweeting
15efb2d5ed new generic_html parser for extracting hrefs 2020-08-18 08:29:05 -04:00
Nick Sweeting
fc0d064ddf add hints 2020-08-18 08:22:34 -04:00
Nick Sweeting
a3155b88a1 better error message when nothing left to resume 2020-08-18 04:42:39 -04:00
Nick Sweeting
f8eb245519 better archivebox schedule CLI UX 2020-08-18 04:39:58 -04:00
Nick Sweeting
ec5609a09a fix resume flag on archivebox update 2020-08-18 04:39:39 -04:00
Nick Sweeting
12f3977da5 fix source filenames missing domain 2020-08-18 04:39:29 -04:00
Nick Sweeting
b681a477ae add overwrite flag to add command to force re-archiving 2020-08-18 04:37:54 -04:00
Nick Sweeting
da671532a4 fix schedule cmd defaulting to day and remove week 2020-08-18 04:35:59 -04:00
Nick Sweeting
235eb20dbd support cron in docker 2020-08-18 01:59:04 -04:00
Nick Sweeting
494be09bc2 add depth flag to schedule cmd 2020-08-18 01:58:54 -04:00
Nick Sweeting
0ef2b17678 only show data locations in version output when in a data dir 2020-08-13 23:21:57 -04:00
Nick Sweeting
33ab7fd4ec autodetect when running inside docker and provide hints 2020-08-10 14:18:04 -04:00
Nick Sweeting
f24cb3dcbe add docker help text 2020-08-10 13:42:31 -04:00
Cristian
a8c74730f8 docs: Add docstring to oneshot method 2020-07-31 10:28:30 -05:00
Cristian
e6c571beb2 fix: Remove title from extractors for oneshot 2020-07-31 10:24:58 -05:00
Cristian
8bcb171e74 fix: Remove support for multiple urls in oneshot command 2020-07-31 09:05:40 -05:00
Cristian
3afb2401bc fix: Add condition to avoid breaking the add command 2020-07-29 11:53:49 -05:00
Cristian
c073ea141d feat: Initial oneshot command proposal 2020-07-29 11:19:06 -05:00
Nick Sweeting
9806ed8d8c fix circular import 2020-07-28 06:50:03 -04:00
Nick Sweeting
b8c93889c1 hide prints and tweak url text in titlebar 2020-07-28 06:03:52 -04:00
Nick Sweeting
9e7330cc14 add init flag to server and fix SHOW_PROGRESS config being ignored 2020-07-28 05:57:34 -04:00
Nick Sweeting
943453a9a8 pass overwrite properly 2020-07-28 05:54:42 -04:00
Nick Sweeting
d6030e15c7 allow passing links to remove method 2020-07-28 05:52:15 -04:00
Nick Sweeting
ea1ff7b6bc fix linter 2020-07-27 23:34:30 -04:00
Nick Sweeting
3aeca0e450 fix pending titles and favicons, improve add page, custom admin 2020-07-27 23:26:45 -04:00
Cristian
6006b4f93b refactor: Organize code to remove flake8 issues 2020-07-24 12:25:25 -05:00
Cristian
fe0884f1ec fix: Remove link from sql index on remove command 2020-07-23 15:07:00 -05:00
apkallum
b854884c56 move umask to init/__config__ 2020-07-23 11:50:42 -04:00
apkallum
0ed2a23670 ensure correct permissions for output folder 2020-07-23 10:28:10 -04:00
Cristian
a5550b2105 fix: Rename logging folder to avoid naming conflicts (and circular import issues) 2020-07-22 11:02:13 -05:00
Cristian
f4d1b5121e refactor: Move logging.py to main module to avoid circular import issues 2020-07-17 18:00:04 -05:00
Cristian
5e2bf73f04 fix: Bugs related to add() refactor 2020-07-13 14:48:25 -05:00
Nick Sweeting
d3bfa98a91 fix depth flag and tweak logging 2020-07-13 11:26:34 -04:00
Nick Sweeting
7cbd068c95 add flake8 2020-07-13 11:22:23 -04:00
Cristian
f12bfeb322 refactor: Change add() to receive url and depth instead of import_str and import_path 2020-07-08 08:17:47 -05:00
Cristian
b68c13918f feat: Disable stdin from archivebox add 2020-07-07 12:39:36 -05:00
Nick Sweeting
f373df7bd4 update helptext to clarify adding links 2020-07-02 13:23:40 -04:00
Nick Sweeting
602e141f08 fix config file atomic writing bugs 2020-06-30 02:04:16 -04:00
Nick Sweeting
44bd6866ac colorize hints in CLI output more consistently 2020-06-30 01:48:41 -04:00
Nick Sweeting
cf01eff9c7 tweak log output text 2020-06-30 01:10:54 -04:00
Nick Sweeting
b9e17fa0d1 rename archivebox-info to archivebox-status 2020-06-25 23:32:01 -04:00
Nick Sweeting
43c471e4af cli experience improvements 2020-06-25 17:47:55 -04:00