148 commits

Author	SHA1	Message	Date
spresse1	603ce7ec10	After a timeout, chrome will leave behind a SingletonLock, which prevents future instances of chrome from starting. When an extractor fails due to a timeout, remove this file.	2023-08-28 17:27:03 +02:00
Sascha Ißbrücker	7bf4f40da0	just use out_dir	2023-05-29 10:03:49 +02:00
Sascha Ißbrücker	40c122515a	fix: make oneshot command return successful exist code	2023-05-29 10:01:27 +02:00
Micah R Ledbetter	1e50ca243e	Add FAVICON_PROVIDER option for custom favicon service	2023-05-05 20:42:36 -05:00
ふぁ	d77c770c47	add CHROME_TIMEOUT args ... Signed-off-by: ふぁ <yuki@yuki0311.com>	2023-03-14 20:29:41 +09:00
Nick Sweeting	9599845b56	ensure DOM HTML dump is non-zero length file when retrying	2023-03-13 10:49:26 +00:00
Nick Sweeting	0cbeeb4346	Merge pull request #1021 from renaisun/dev	2023-01-09 18:17:39 -08:00
Joseph Turian	07de4a79a1	Merge branch 'dev' into feature/kludge-984-UTF8-bug	2022-12-20 11:39:01 +01:00
Joseph Turian	081a12b079	Add ts	2022-09-12 21:32:47 +00:00
Joseph Turian	daef48e59b	flake8	2022-09-12 21:31:33 +00:00
Joseph Turian	983f485cc0	flake8	2022-09-12 21:29:43 +00:00
Joseph Turian	b864c38d9e	Don't be strict on unicode errors	2022-09-12 20:40:45 +00:00
Joseph Turian	dba423a568	A few more youtube-dl tweaks	2022-09-12 20:36:23 +00:00
Joseph Turian	f5f7aff3b4	Added yt-dlp everywhere	2022-09-12 20:34:02 +00:00
renaisun	0ea955b3ed	add a missing comma	2022-09-12 09:08:28 +08:00
notevenaperson	40659b5e9d	singlefile.py: Code to ensure options are deduplicated	2022-09-12 09:08:28 +08:00
Joseph Turian	2b58cce43f	Attempted to warn on #984 and #1014	2022-09-11 12:19:16 +02:00
renaisun	8899fe0b92	Add SINGLEFILE_ARGS to control single-file arguments	2022-06-09 14:35:48 +08:00
Nick Sweeting	950b5cbbb6	Merge pull request #924 from prnake/dev ... improve title extractor	2022-05-09 18:38:12 -07:00
Nick Sweeting	57df65f28f	use yt-dlp for media archiving instead of youtube-dl	2022-04-21 07:11:35 -07:00
prnake	011bd104cb	remove unused import	2022-02-09 10:48:51 +08:00
papersnake	de8e22efb7	improve title extractor	2022-02-08 23:17:52 +08:00
Nick Sweeting	4715ace7dd	ignore BaseException lgtm errors	2021-05-31 20:59:05 -04:00
Nick Sweeting	eb4d3bca9d	Update readability.py	2021-05-13 00:13:32 -04:00
Nick Sweeting	62078a77f8	show run duration after each archived link in cli output	2021-04-10 07:52:01 -04:00
Nick Sweeting	193df5c8d3	add video subtitles and description to full-text index	2021-04-10 07:22:20 -04:00
Nick Sweeting	a9986f1f05	add timezone support, tons of CSS and layout improvements, more detailed snapshot admin form info, ability to sort by recently updated, better grid view styling, better table layouts, better dark mode support	2021-04-10 04:21:36 -04:00
Nick Sweeting	bd6d9c165b	enforce utf8 on literally all file operations because windows sucks	2021-03-27 01:16:29 -04:00
Nick Sweeting	084cf7ff51	add more explanation about snapshot.save timestamp bump	2021-02-17 13:34:46 -05:00
Nick Sweeting	acb932ba12	improve readability and mercury error handling and fix output path to be relative	2021-02-16 15:53:11 -05:00
Nick Sweeting	c95698e608	bump Snapshot.updated time after each extractor, change extractor order	2021-02-16 15:52:18 -05:00
Nick Sweeting	d0f8a5e710	change mercury atomic_write output order	2021-02-16 06:19:16 -05:00
Nick Sweeting	7d0f5653c3	fix lgtm alerts	2021-02-01 02:27:24 -05:00
Nick Sweeting	04c951cdd5	fix alerts	2021-02-01 02:22:02 -05:00
Nick Sweeting	846c966c4d	use globbing to find wget output path	2021-01-30 22:02:39 -05:00
Nick Sweeting	e6fa16e13a	only chmod wget output if it exists	2021-01-30 22:02:11 -05:00
Nick Sweeting	385daf9af8	save the url as title for staticfiles or non html files	2021-01-30 22:01:49 -05:00
Nick Sweeting	b9b1c3d9e8	fix singlefile output path not relative	2021-01-30 20:44:49 -05:00
Nick Sweeting	d6de04a83a	fix lgtm errors	2021-01-30 06:07:35 -05:00
Nick Sweeting	c2aaa41c76	fix missing str path	2021-01-30 01:25:08 -05:00
Nick Sweeting	15e58bd366	fix using os.path calls on pathlib paths	2021-01-27 11:27:40 -05:00
Nick Sweeting	9764a8ed9b	check for non html files from wget	2021-01-25 18:15:16 -05:00
Dan Arnfield	5420903102	Refactor should_save_extractor methods to accept overwrite parameter	2021-01-21 15:56:32 -06:00
Nick Sweeting	ef7711ffa0	fix cookies file arg is path	2021-01-20 19:13:53 -05:00
Cristian	6031ffa3b2	fix: Mercury extractor error was incorrectly initialized	2021-01-07 09:22:46 -05:00
Cristian	e9e4adfc34	fix: wget_output_path failing on some extractors. Add a new condition	2021-01-07 09:07:29 -05:00
Cristian	81d766aba1	refactor: Remove setup_django from title.py	2020-12-11 16:03:50 -05:00
Cristian	275ad22db7	refactor: Remove skip_index from archive related functions	2020-12-08 18:42:25 -05:00
Cristian	f6c73f9aeb	fix: Issue with oneshot command	2020-12-08 18:42:25 -05:00
JDC	7903db6dfb	Add ArchiveResult Manager and sorted indexable filter	2020-12-06 01:13:39 +02:00