206 commits

Author	SHA1	Message	Date
Cristian	a28547cbca	refactor: Remove get_empty_snapshot queryset function and generate it directly	2020-12-11 16:27:15 -05:00
Nick Sweeting	e90cf05141	fix lint errors	2020-12-11 16:51:11 +02:00
Nick Sweeting	30f8d3f191	show python implementation name and flip verison output order for easier reading when wrapped on small screens	2020-12-11 16:21:52 +02:00
Nick Sweeting	9fa70b3452	add extractors arg to oneshot command and bump version to v0.5.1	2020-12-11 15:48:46 +02:00
Cristian	275ad22db7	refactor: Remove skip_index from archive related functions	2020-12-08 18:42:25 -05:00
Cristian	9aac09a5e1	feat: Patch setup_django so we can use an inmemory db in specific commands	2020-12-08 18:42:25 -05:00
JDC	caf4660ac8	Add indexing to update command and utilities	2020-12-06 01:14:37 +02:00
JDC	c5b1b91708	fix: flush_search_index must be called before removing snapshots	2020-12-06 01:13:39 +02:00
JDC	823df34080	Use QuerySets for search backend API instead of pks	2020-12-06 01:12:47 +02:00
JDC	f383648ffc	Use a generator for snapshot flush from index	2020-12-06 01:12:47 +02:00
JDC	47daa038eb	Implement flush for search backend after remove command	2020-12-06 01:12:47 +02:00
Nick Sweeting	3b280e6b02	Merge pull request #569 from cdvv7788/extract-command-update ... feat: Add --extract flag to update command	2020-12-05 17:43:28 -05:00
Cristian	35389608d1	feat: Add --extract flag to update command	2020-12-05 12:20:47 -05:00
Cristian	34cad4fe8d	fix: Update function with --index-only flag was not behaving as expected	2020-12-05 12:10:17 -05:00
Cristian	7008f9b735	feat: move import	2020-11-28 13:11:15 -05:00
Cristian	fa5de72f9f	refactor: Move indexing logic out of logging module	2020-11-28 12:34:40 -05:00
Nick Sweeting	c9162a6d09	remove finished/not finished spinners	2020-11-28 01:07:02 -05:00
Nick Sweeting	1b22f8eeef	Merge pull request #515 from cdvv7788/POC-setup-django-on-init	2020-11-27 23:56:37 -05:00
Nick Sweeting	e4d2ac432d	improve OS kernel output in archivebox version	2020-11-27 23:08:23 -05:00
Nick Sweeting	07a56f9d46	also print platform and CPU info in version output	2020-11-27 22:59:18 -05:00
Nick Sweeting	d9ef3d0bf8	ignore lost+found dir in data folder	2020-11-27 19:39:19 -05:00
mAAdhaTTah	be7a7f8548	Fix string checks in schedule ... `s` comes through as a `PosixPath`, so both the `' ' in s` & return value, later used by `join`, complain.	2020-11-23 18:34:07 -05:00
Nick Sweeting	0e2ccbc10d	update urls to new repo path	2020-11-23 02:06:46 -05:00
Nick Sweeting	b11d562445	fix splitting on multiple equals in val	2020-11-22 12:33:15 -05:00
Cristian	54df0a035b	fix: Move csv split to the add function to avoid optional nullable argument	2020-11-13 13:10:17 -05:00
Cristian	db523c9d82	fix: Avoid mutable default input argument	2020-11-13 11:41:50 -05:00
Cristian	44eede96e5	feat: Add extract flag to add command	2020-11-13 09:24:34 -05:00
Nick Sweeting	79051ca15b	new package build	2020-10-31 03:08:41 -04:00
Nick Sweeting	18355dc2c6	clean up config loading in settings and config file layout	2020-10-31 03:08:03 -04:00
Cristian	e1d0b8bce7	feat: Initialize django at the beginning	2020-10-26 07:45:21 -05:00
Cristian	f397634dd2	feat: Rename old indexes at the end of init process	2020-10-23 06:45:56 -05:00
Cristian	ae1484b8bf	feat: Remove index.json and index.html generation from the regular process	2020-10-23 06:45:56 -05:00
Cristian	b18bbf8874	test: Fix tests post-rebase	2020-09-17 09:09:52 -05:00
apkallum	594d9e49ce	first attempt to migrate to Pathlib	2020-09-17 09:09:52 -05:00
Cristian	5348f4735a	fix: Change check to avoid issues with empty querysets	2020-09-15 08:05:46 -05:00
Cristian	cf18130f85	feat: Add deprecation warning for index.json	2020-09-15 08:05:46 -05:00
Cristian	fa622d3e14	refactor: Replace --index with --with-headers in the list command to make it more explicit. Change it so it affects the csv output too.	2020-09-15 08:05:46 -05:00
Cristian	2aa8d69b72	fix: Save history in main index (to mimic previous behaviour)	2020-09-15 08:05:46 -05:00
Cristian	7e9d195d13	feat: Update list command to sort using sqlite	2020-09-15 08:05:46 -05:00
Cristian	f55153eab3	feat: Update update command to work with querysets	2020-09-15 08:05:46 -05:00
Cristian	a8ed72501d	feat: Refactor remove command to use querysets	2020-09-15 08:05:46 -05:00
Cristian	be520d137a	feat: Refactor add method to use querysets	2020-09-15 08:05:46 -05:00
Cristian	6a2e6aad2f	fix: status command was failing on empty archives	2020-09-15 08:05:46 -05:00
Cristian	be0dff8126	feat: Add tests to refactored init command	2020-09-15 08:05:46 -05:00
Cristian	404f333e17	feat: Refactor get_invalid_folders to work with a queryset instead of a list of links	2020-09-15 08:05:46 -05:00
Cristian	dae606de6e	feat: Update init to take advantage of querysets to reduce memory consumption	2020-09-15 08:05:46 -05:00
Cristian	6b4b7127b4	feat: Remove unused imports	2020-09-15 08:05:46 -05:00
Cristian	b8585dd92e	feat: load_main_index returns a queryset now	2020-09-15 08:05:46 -05:00
Cristian	885ff50449	feat: Add html export to list command	2020-09-15 08:05:46 -05:00
Cristian	aab8f96520	feat: Add flag to list command to support index like output	2020-09-15 08:05:46 -05:00
Cristian	be57db1369	feat: Save static indexes at the end of init	2020-09-15 08:05:46 -05:00
Cristian	c16fdf1b47	feat: Update data folder check	2020-09-15 08:05:46 -05:00
Cristian	874403e667	feat: Remove patch_main_index	2020-09-15 08:05:46 -05:00
Cristian	31343c1367	feat: Update extractors and add command to use sql index as source of truth	2020-09-15 08:05:46 -05:00
Cristian	e9caee6b10	feat: Update status command to consider sql as the main index	2020-09-15 08:05:46 -05:00
Cristian	02f36b2096	feat: Replace index.json with index.sql as the main index in init	2020-09-15 08:05:46 -05:00
Nick Sweeting	5cb13ff1dd	ignore node cruft in output folder during init	2020-08-18 18:20:26 -04:00
Nick Sweeting	15efb2d5ed	new generic_html parser for extracting hrefs	2020-08-18 08:29:05 -04:00
Nick Sweeting	fc0d064ddf	add hints	2020-08-18 08:22:34 -04:00
Nick Sweeting	a3155b88a1	better error message when nothing left to resume	2020-08-18 04:42:39 -04:00
Nick Sweeting	f8eb245519	better archivebox schedule CLI UX	2020-08-18 04:39:58 -04:00
Nick Sweeting	ec5609a09a	fix resume flag on archivebox update	2020-08-18 04:39:39 -04:00
Nick Sweeting	12f3977da5	fix source filenames missing domain	2020-08-18 04:39:29 -04:00
Nick Sweeting	b681a477ae	add overwrite flag to add command to force re-archiving	2020-08-18 04:37:54 -04:00
Nick Sweeting	da671532a4	fix schedule cmd defaulting to day and remove week	2020-08-18 04:35:59 -04:00
Nick Sweeting	235eb20dbd	support cron in docker	2020-08-18 01:59:04 -04:00
Nick Sweeting	494be09bc2	add depth flag to schedule cmd	2020-08-18 01:58:54 -04:00
Nick Sweeting	0ef2b17678	only show data locations in version output when in a data dir	2020-08-13 23:21:57 -04:00
Nick Sweeting	33ab7fd4ec	autodetect when running inside docker and provide hints	2020-08-10 14:18:04 -04:00
Nick Sweeting	f24cb3dcbe	add docker help text	2020-08-10 13:42:31 -04:00
Cristian	a8c74730f8	docs: Add docstring to oneshot method	2020-07-31 10:28:30 -05:00
Cristian	e6c571beb2	fix: Remove title from extractors for oneshot	2020-07-31 10:24:58 -05:00
Cristian	8bcb171e74	fix: Remove support for multiple urls in oneshot command	2020-07-31 09:05:40 -05:00
Cristian	3afb2401bc	fix: Add condition to avoid breaking the add command	2020-07-29 11:53:49 -05:00
Cristian	c073ea141d	feat: Initial oneshot command proposal	2020-07-29 11:19:06 -05:00
Nick Sweeting	9806ed8d8c	fix circular import	2020-07-28 06:50:03 -04:00
Nick Sweeting	b8c93889c1	hide prints and tweak url text in titlebar	2020-07-28 06:03:52 -04:00
Nick Sweeting	9e7330cc14	add init flag to server and fix SHOW_PROGRESS config being ignored	2020-07-28 05:57:34 -04:00
Nick Sweeting	943453a9a8	pass overwrite properly	2020-07-28 05:54:42 -04:00
Nick Sweeting	d6030e15c7	allow passing links to remove method	2020-07-28 05:52:15 -04:00
Nick Sweeting	ea1ff7b6bc	fix linter	2020-07-27 23:34:30 -04:00
Nick Sweeting	3aeca0e450	fix pending titles and favicons, improve add page, custom admin	2020-07-27 23:26:45 -04:00
Cristian	6006b4f93b	refactor: Organize code to remove flake8 issues	2020-07-24 12:25:25 -05:00
Cristian	fe0884f1ec	fix: Remove link from sql index on remove command	2020-07-23 15:07:00 -05:00
apkallum	b854884c56	move umask to init/__config__	2020-07-23 11:50:42 -04:00
apkallum	0ed2a23670	ensure correct permissions for output folder	2020-07-23 10:28:10 -04:00
Cristian	a5550b2105	fix: Rename logging folder to avoid naming conflicts (and circular import issues)	2020-07-22 11:02:13 -05:00
Cristian	f4d1b5121e	refactor: Move logging.py to main module to avoid circular import issues	2020-07-17 18:00:04 -05:00
Cristian	5e2bf73f04	fix: Bugs related to add() refactor	2020-07-13 14:48:25 -05:00
Nick Sweeting	d3bfa98a91	fix depth flag and tweak logging	2020-07-13 11:26:34 -04:00
Nick Sweeting	7cbd068c95	add flake8	2020-07-13 11:22:23 -04:00
Cristian	f12bfeb322	refactor: Change add() to receive url and depth instead of import_str and import_path	2020-07-08 08:17:47 -05:00
Cristian	b68c13918f	feat: Disable stdin from archivebox add	2020-07-07 12:39:36 -05:00
Nick Sweeting	f373df7bd4	update helptext to clarify adding links	2020-07-02 13:23:40 -04:00
Nick Sweeting	602e141f08	fix config file atomic writing bugs	2020-06-30 02:04:16 -04:00
Nick Sweeting	44bd6866ac	colorize hints in CLI output more consistently	2020-06-30 01:48:41 -04:00
Nick Sweeting	cf01eff9c7	tweak log output text	2020-06-30 01:10:54 -04:00
Nick Sweeting	b9e17fa0d1	rename archivebox-info to archivebox-status	2020-06-25 23:32:01 -04:00
Nick Sweeting	43c471e4af	cli experience improvements	2020-06-25 17:47:55 -04:00
Nick Sweeting	3c3b2ee621	expose more django server config options	2019-05-02 19:15:16 -04:00
Nick Sweeting	ad3898add6	allow forcing init in dirty directory	2019-05-01 02:27:50 -04:00
Nick Sweeting	95007d9137	split up utils into separate files	2019-04-30 23:13:04 -04:00
Nick Sweeting	daf5951897	allow passing debug flag to archivebox server	2019-04-30 23:11:41 -04:00
Nick Sweeting	158f145d9a	move docstrings to main.py out of cli files	2019-04-30 23:10:48 -04:00
Nick Sweeting	248a4da99d	fix docs to use read-the-docs theme	2019-04-27 17:43:14 -04:00
Nick Sweeting	1b8abc0961	move everything out of legacy folder	2019-04-27 17:26:24 -04:00