200 commits

Author	SHA1	Message	Date
Nick Sweeting	8cfe6f4afb	cleanup update flag handling and show better logging to clarify when its working	2022-05-09 20:15:55 -07:00
Nick Sweeting	38e54b93fe	allow parsing to continue even when fetching URL contents fails	2022-05-09 19:56:24 -07:00
Ross	c63822a5e5	Fix missing input redirection in a hint text	2022-04-19 22:25:49 +01:00
Nick Sweeting	e0a2b2e252	ominous warnings	2021-06-01 03:03:42 -04:00
Nick Sweeting	4715ace7dd	ignore BaseException lgtm errors	2021-05-31 20:59:05 -04:00
Nick Sweeting	3db77fd1a2	fix lint errors	2021-04-24 03:49:01 -04:00
Nick Sweeting	fa84f749ff	run archivebox version using python binary	2021-04-24 03:29:22 -04:00
Nick Sweeting	226e26852c	dont try to autodetect whether node, chrome, etc are needed during setup	2021-04-24 03:29:22 -04:00
Nick Sweeting	f129b9f443	allow executing main	2021-04-24 03:11:06 -04:00
Nick Sweeting	4f91af0ce9	TODO create admin user from env vars	2021-04-23 15:50:34 -04:00
Nick Sweeting	63552be0e3	whoops	2021-04-13 04:03:47 -04:00
Nick Sweeting	7d1f8cb496	no longer recommend snap chromium, just use playwright	2021-04-13 03:59:17 -04:00
Nick Sweeting	54cab113f3	allow static_index.json in dir root from old static exports	2021-04-12 17:06:32 -04:00
Nick Sweeting	a9986f1f05	add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support	2021-04-10 04:21:36 -04:00
Nick Sweeting	cadac4838e	fix searching by url contents	2021-04-05 23:21:07 -04:00
Nick Sweeting	2fc9f1cfa3	dont depend on README in Dockerfile install step	2021-04-05 23:17:07 -04:00
Nick Sweeting	8b4f84959a	new setup flag on init command to autosetup on init	2021-04-05 21:15:32 -04:00
Nick Sweeting	5c181532b5	only install dependencies in setup if they are missing	2021-04-05 21:15:10 -04:00
Nick Sweeting	79fb60a47a	dont allow caching when installing packages	2021-04-05 20:08:48 -04:00
Nick Sweeting	8c1d12020f	fix lint errors	2021-04-05 19:53:42 -04:00
Nick Sweeting	3449bac5bc	add new setup command to autoinstall dependencies	2021-04-05 19:51:57 -04:00
Nick Sweeting	24f7dd9c05	dont raise systemexit on succesful config	2021-04-05 19:51:40 -04:00
Nick Sweeting	fc5d99b4b4	fix missing package.json repo	2021-04-01 01:40:02 -04:00
Nick Sweeting	7ee7a94e67	fix str split using old overwrite pipe instead of append pipe	2021-03-31 11:30:15 -04:00
Nick Sweeting	005da31f8f	add overwrite flag option to schedule cmd	2021-03-31 11:29:51 -04:00
Nick Sweeting	534ae60ab0	dont overwrite the log on each run, and use dedicated log for scheduled archiving	2021-03-31 11:20:42 -04:00
Nick Sweeting	36f0646501	Merge pull request #669 from FliegendeWurst/fix-issue-235 ... add command: --parser option (fixes #235)	2021-03-31 00:53:47 -04:00
FliegendeWurst	60bd9a902e	add command: --parser option	2021-03-28 10:09:11 +02:00
Nick Sweeting	49b77b91bb	fix unused var	2021-03-27 05:12:54 -04:00
Nick Sweeting	f3a3d76439	support adding urls with tags directly via CLI and add page	2021-03-27 04:30:15 -04:00
Nick Sweeting	fea0b89dbe	add tag cli option	2021-03-27 03:57:05 -04:00
Nick Sweeting	ba2afa0672	allow cancelling init partway through and show hints to resume	2021-02-18 02:34:42 -05:00
Nick Sweeting	bdf1b102be	load ArchiveResults from orphaned links history during init	2021-02-16 06:20:05 -05:00
Nick Sweeting	fad2620c62	minor tweaks	2021-02-16 02:49:31 -05:00
Nick Sweeting	4eac9d0ec1	show extra system setup info in version output	2021-02-16 01:23:58 -05:00
Nick Sweeting	b3a50a2c10	fix server quick-init param not being passed properly to subcommand	2021-02-15 20:49:40 -05:00
Nick Sweeting	b06e256ad9	fix add command not updating snapshot detail index pages when passed index-only and overwrite flags together	2021-02-15 20:49:23 -05:00
Nick Sweeting	8e98cef7ad	fix after and before args flipped when filtering	2021-02-15 20:48:51 -05:00
Nick Sweeting	00ae1f15a7	ignore shm db file and config files in archivebox data dir on init	2021-02-15 14:52:37 -05:00
Nick Sweeting	3c3bae02d2	add quick-init option to skip reimporting all snapshot dirs on init	2021-02-15 14:52:10 -05:00
Nick Sweeting	611216765d	switch sqlite to use WAL mode by default to prevent database locked errors	2021-02-15 13:39:03 -05:00
Nick Sweeting	783f597955	minor build fixes	2021-02-01 05:13:46 -05:00
Nick Sweeting	ed13ec7655	remove active theme	2021-01-30 05:47:55 -05:00
Nick Sweeting	39ec77e46c	add createsuperuser flag to server command	2021-01-28 22:27:02 -05:00
Cristian	a28547cbca	refactor: Remove get_empty_snapshot queryset function and generate it directly	2020-12-11 16:27:15 -05:00
Nick Sweeting	e90cf05141	fix lint errors	2020-12-11 16:51:11 +02:00
Nick Sweeting	30f8d3f191	show python implementation name and flip verison output order for easier reading when wrapped on small screens	2020-12-11 16:21:52 +02:00
Nick Sweeting	9fa70b3452	add extractors arg to oneshot command and bump version to v0.5.1	2020-12-11 15:48:46 +02:00
Cristian	275ad22db7	refactor: Remove skip_index from archive related functions	2020-12-08 18:42:25 -05:00
Cristian	9aac09a5e1	feat: Patch setup_django so we can use an inmemory db in specific commands	2020-12-08 18:42:25 -05:00
JDC	caf4660ac8	Add indexing to update command and utilities	2020-12-06 01:14:37 +02:00
JDC	c5b1b91708	fix: flush_search_index must be called before removing snapshots	2020-12-06 01:13:39 +02:00
JDC	823df34080	Use QuerySets for search backend API instead of pks	2020-12-06 01:12:47 +02:00
JDC	f383648ffc	Use a generator for snapshot flush from index	2020-12-06 01:12:47 +02:00
JDC	47daa038eb	Implement flush for search backend after remove command	2020-12-06 01:12:47 +02:00
Nick Sweeting	3b280e6b02	Merge pull request #569 from cdvv7788/extract-command-update ... feat: Add --extract flag to update command	2020-12-05 17:43:28 -05:00
Cristian	35389608d1	feat: Add --extract flag to update command	2020-12-05 12:20:47 -05:00
Cristian	34cad4fe8d	fix: Update function with --index-only flag was not behaving as expected	2020-12-05 12:10:17 -05:00
Cristian	7008f9b735	feat: move import	2020-11-28 13:11:15 -05:00
Cristian	fa5de72f9f	refactor: Move indexing logic out of logging module	2020-11-28 12:34:40 -05:00
Nick Sweeting	c9162a6d09	remove finished/not finished spinners	2020-11-28 01:07:02 -05:00
Nick Sweeting	1b22f8eeef	Merge pull request #515 from cdvv7788/POC-setup-django-on-init	2020-11-27 23:56:37 -05:00
Nick Sweeting	e4d2ac432d	improve OS kernel output in archivebox version	2020-11-27 23:08:23 -05:00
Nick Sweeting	07a56f9d46	also print platform and CPU info in version output	2020-11-27 22:59:18 -05:00
Nick Sweeting	d9ef3d0bf8	ignore lost+found dir in data folder	2020-11-27 19:39:19 -05:00
mAAdhaTTah	be7a7f8548	Fix string checks in schedule ... `s` comes through as a `PosixPath`, so both the `' ' in s` & return value, later used by `join`, complain.	2020-11-23 18:34:07 -05:00
Nick Sweeting	0e2ccbc10d	update urls to new repo path	2020-11-23 02:06:46 -05:00
Nick Sweeting	b11d562445	fix splitting on multiple equals in val	2020-11-22 12:33:15 -05:00
Cristian	54df0a035b	fix: Move csv split to the add function to avoid optional nullable argument	2020-11-13 13:10:17 -05:00
Cristian	db523c9d82	fix: Avoid mutable default input argument	2020-11-13 11:41:50 -05:00
Cristian	44eede96e5	feat: Add extract flag to add command	2020-11-13 09:24:34 -05:00
Nick Sweeting	79051ca15b	new package build	2020-10-31 03:08:41 -04:00
Nick Sweeting	18355dc2c6	clean up config loading in settings and config file layout	2020-10-31 03:08:03 -04:00
Cristian	e1d0b8bce7	feat: Initialize django at the beginning	2020-10-26 07:45:21 -05:00
Cristian	f397634dd2	feat: Rename old indexes at the end of init process	2020-10-23 06:45:56 -05:00
Cristian	ae1484b8bf	feat: Remove index.json and index.html generation from the regular process	2020-10-23 06:45:56 -05:00
Cristian	b18bbf8874	test: Fix tests post-rebase	2020-09-17 09:09:52 -05:00
apkallum	594d9e49ce	first attempt to migrate to Pathlib	2020-09-17 09:09:52 -05:00
Cristian	5348f4735a	fix: Change check to avoid issues with empty querysets	2020-09-15 08:05:46 -05:00
Cristian	cf18130f85	feat: Add deprecation warning for index.json	2020-09-15 08:05:46 -05:00
Cristian	fa622d3e14	refactor: Replace --index with --with-headers in the list command to make it more explicit. Change it so it affects the csv output too.	2020-09-15 08:05:46 -05:00
Cristian	2aa8d69b72	fix: Save history in main index (to mimic previous behaviour)	2020-09-15 08:05:46 -05:00
Cristian	7e9d195d13	feat: Update list command to sort using sqlite	2020-09-15 08:05:46 -05:00
Cristian	f55153eab3	feat: Update update command to work with querysets	2020-09-15 08:05:46 -05:00
Cristian	a8ed72501d	feat: Refactor remove command to use querysets	2020-09-15 08:05:46 -05:00
Cristian	be520d137a	feat: Refactor add method to use querysets	2020-09-15 08:05:46 -05:00
Cristian	6a2e6aad2f	fix: status command was failing on empty archives	2020-09-15 08:05:46 -05:00
Cristian	be0dff8126	feat: Add tests to refactored init command	2020-09-15 08:05:46 -05:00
Cristian	404f333e17	feat: Refactor get_invalid_folders to work with a queryset instead of a list of links	2020-09-15 08:05:46 -05:00
Cristian	dae606de6e	feat: Update init to take advantage of querysets to reduce memory consumption	2020-09-15 08:05:46 -05:00
Cristian	6b4b7127b4	feat: Remove unused imports	2020-09-15 08:05:46 -05:00
Cristian	b8585dd92e	feat: load_main_index returns a queryset now	2020-09-15 08:05:46 -05:00
Cristian	885ff50449	feat: Add html export to list command	2020-09-15 08:05:46 -05:00
Cristian	aab8f96520	feat: Add flag to list command to support index like output	2020-09-15 08:05:46 -05:00
Cristian	be57db1369	feat: Save static indexes at the end of init	2020-09-15 08:05:46 -05:00
Cristian	c16fdf1b47	feat: Update data folder check	2020-09-15 08:05:46 -05:00
Cristian	874403e667	feat: Remove patch_main_index	2020-09-15 08:05:46 -05:00
Cristian	31343c1367	feat: Update extractors and add command to use sql index as source of truth	2020-09-15 08:05:46 -05:00
Cristian	e9caee6b10	feat: Update status command to consider sql as the main index	2020-09-15 08:05:46 -05:00
Cristian	02f36b2096	feat: Replace index.json with index.sql as the main index in init	2020-09-15 08:05:46 -05:00