547 commits

Author	SHA1	Message	Date
Nick Sweeting	c9b3bab84d	fix pull title not working	2020-08-18 08:49:26 -04:00
Nick Sweeting	f18d92570e	wip attempt to fix timestamp unique constraint errors	2020-08-18 08:30:09 -04:00
Nick Sweeting	b0c0a676f8	re-enable readability and singlefile by default now that its less noisy	2020-08-18 08:29:46 -04:00
Nick Sweeting	15efb2d5ed	new generic_html parser for extracting hrefs	2020-08-18 08:29:05 -04:00
Nick Sweeting	a682a9c478	make all parsers accept arbitrary meta kwargs	2020-08-18 08:27:47 -04:00
Nick Sweeting	2e2b4f8150	fix url is too long to be a path error	2020-08-18 08:23:57 -04:00
Nick Sweeting	e3ac4c2405	htmldecode downloaded sources before parsing for links	2020-08-18 08:23:20 -04:00
Nick Sweeting	fc0d064ddf	add hints	2020-08-18 08:22:34 -04:00
Nick Sweeting	9d093a34f8	better dependency version check	2020-08-18 08:21:55 -04:00
Nick Sweeting	2f282402c7	add hint and prefix stdout utils	2020-08-18 08:21:36 -04:00
Nick Sweeting	2557a0da04	tweak cli output coloring	2020-08-18 08:16:18 -04:00
Nick Sweeting	d7d53cfb12	dont show skipped extractors to reduce visual noise	2020-08-18 08:13:35 -04:00
Nick Sweeting	5f84a7bc6e	better handle the case where json index lags behind sql index	2020-08-18 08:13:13 -04:00
Nick Sweeting	77d2f08a5c	show more info in merge conflict error message	2020-08-18 08:12:35 -04:00
Nick Sweeting	718d39e242	add common code extensions to default blacklist	2020-08-18 08:12:10 -04:00
Nick Sweeting	3658153cf8	fix url parsing through quotes	2020-08-18 08:04:57 -04:00
Nick Sweeting	a3155b88a1	better error message when nothing left to resume	2020-08-18 04:42:39 -04:00
Nick Sweeting	f8eb245519	better archivebox schedule CLI UX	2020-08-18 04:39:58 -04:00
Nick Sweeting	ec5609a09a	fix resume flag on archivebox update	2020-08-18 04:39:39 -04:00
Nick Sweeting	12f3977da5	fix source filenames missing domain	2020-08-18 04:39:29 -04:00
Nick Sweeting	ec1cce283e	include failed link updates in total count stats	2020-08-18 04:39:19 -04:00
Nick Sweeting	6e142e3531	match existing snapshots on url only, ignoring timestamp	2020-08-18 04:39:04 -04:00
Nick Sweeting	f371032b71	show warning when killing archivebox during index writing	2020-08-18 04:38:29 -04:00
Nick Sweeting	92de20af15	better detect missing dependencies on startup	2020-08-18 04:38:13 -04:00
Nick Sweeting	b681a477ae	add overwrite flag to add command to force re-archiving	2020-08-18 04:37:54 -04:00
Nick Sweeting	da671532a4	fix schedule cmd defaulting to day and remove week	2020-08-18 04:35:59 -04:00
Nick Sweeting	84a8b0d713	hide progress bars on mac	2020-08-18 03:27:52 -04:00
Nick Sweeting	29c8da83d6	0.4.16 release	2020-08-18 02:08:52 -04:00
Nick Sweeting	7c16944a44	Merge pull request #446 from cdvv7788/hotfix/#445	2020-08-18 02:06:32 -04:00
Nick Sweeting	7638dc45ea	0.4.15 release	2020-08-18 01:59:50 -04:00
Nick Sweeting	235eb20dbd	support cron in docker	2020-08-18 01:59:04 -04:00
Nick Sweeting	494be09bc2	add depth flag to schedule cmd	2020-08-18 01:58:54 -04:00
Cristian	05c71fc302	fix: Organize readability extractor so a timeout does not break the whole process	2020-08-17 08:34:40 -05:00
Nick Sweeting	225b63b732	skip invalid urls at all stages	2020-08-17 03:12:17 -04:00
Nick Sweeting	429f39dec1	0.4.14 release	2020-08-14 13:13:50 -04:00
Nick Sweeting	58e928520a	tweak log output for skipped methods	2020-08-14 13:12:50 -04:00
Nick Sweeting	03b73bfe77	Update archivebox/extractors/readability.py	2020-08-14 12:55:22 -04:00
Nick Sweeting	050b717bb9	Merge branch 'master' into readability-extractor	2020-08-14 12:35:35 -04:00
Nick Sweeting	0ef2b17678	only show data locations in version output when in a data dir	2020-08-13 23:21:57 -04:00
Nick Sweeting	a0901ba474	use BIND_ADDR config default for runserver	2020-08-13 23:21:37 -04:00
Cristian	b7aa3df8d2	feat: Disable singlefile and readability by default	2020-08-12 14:42:21 -05:00
Cristian	eb3528fa9f	feat: Add readability output to legacy index.html	2020-08-11 12:14:13 -05:00
Cristian	5dc7e63792	feat: Update dockerfile to support readability	2020-08-11 11:52:43 -05:00
Cristian	2a68af1b94	tests: Add readability tests	2020-08-11 11:15:15 -05:00
Cristian	8aa7b34de7	tests: Add readability to ignored methods in tests	2020-08-11 08:58:49 -05:00
Cristian	dc87d8b68c	tests: Update failing tests	2020-08-11 08:48:13 -05:00
Cristian	0ec747f64e	feat: Look in wget, singlefile or dom outputs before attempting to download the information again	2020-08-11 08:37:12 -05:00
Cristian	a14762640e	feat: Avoid running readability when the target is a file	2020-08-11 08:37:12 -05:00
Cristian	61e08a7c43	docs: Update docs link	2020-08-11 08:37:12 -05:00
Cristian	b33c66a9f7	feat: Split output of readability into multiple files	2020-08-11 08:37:12 -05:00