Commit graph

119 commits

Author SHA1 Message Date
Nick Sweeting
acb932ba12 improve readability and mercury error handling and fix output path to be relative 2021-02-16 15:53:11 -05:00
Nick Sweeting
c95698e608 bump Snapshot.updated time after each extractor, change extractor order 2021-02-16 15:52:18 -05:00
Nick Sweeting
d0f8a5e710 change mercury atomic_write output order 2021-02-16 06:19:16 -05:00
Nick Sweeting
7d0f5653c3 fix lgtm alerts 2021-02-01 02:27:24 -05:00
Nick Sweeting
04c951cdd5 fix alerts 2021-02-01 02:22:02 -05:00
Nick Sweeting
846c966c4d use globbing to find wget output path 2021-01-30 22:02:39 -05:00
Nick Sweeting
e6fa16e13a only chmod wget output if it exists 2021-01-30 22:02:11 -05:00
Nick Sweeting
385daf9af8 save the url as title for staticfiles or non html files 2021-01-30 22:01:49 -05:00
Nick Sweeting
b9b1c3d9e8 fix singlefile output path not relative 2021-01-30 20:44:49 -05:00
Nick Sweeting
d6de04a83a fix lgtm errors 2021-01-30 06:07:35 -05:00
Nick Sweeting
c2aaa41c76 fix missing str path 2021-01-30 01:25:08 -05:00
Nick Sweeting
15e58bd366 fix using os.path calls on pathlib paths 2021-01-27 11:27:40 -05:00
Nick Sweeting
9764a8ed9b check for non html files from wget 2021-01-25 18:15:16 -05:00
Dan Arnfield
5420903102 Refactor should_save_extractor methods to accept overwrite parameter 2021-01-21 15:56:32 -06:00
Nick Sweeting
ef7711ffa0 fix cookies file arg is path 2021-01-20 19:13:53 -05:00
Cristian
6031ffa3b2 fix: Mercury extractor error was incorrectly initialized 2021-01-07 09:22:46 -05:00
Cristian
e9e4adfc34 fix: wget_output_path failing on some extractors. Add a new condition 2021-01-07 09:07:29 -05:00
Cristian
81d766aba1 refactor: Remove setup_django from title.py 2020-12-11 16:03:50 -05:00
Cristian
275ad22db7 refactor: Remove skip_index from archive related functions 2020-12-08 18:42:25 -05:00
Cristian
f6c73f9aeb fix: Issue with oneshot command 2020-12-08 18:42:25 -05:00
JDC
7903db6dfb Add ArchiveResult Manager and sorted indexable filter 2020-12-06 01:13:39 +02:00
JDC
b1f70b2197 Initial implementation 2020-12-06 01:12:45 +02:00
Cristian
33182fd53c fix: Add missing assignation 2020-11-04 15:07:45 -05:00
Cristian
d064a3eeff fix: Handle case when update tries to re-add a link that is not in the sql index 2020-11-04 15:02:54 -05:00
Cristian
f292cface2 fix: Add condition for oneshot when archiving links 2020-11-04 14:40:44 -05:00
Cristian
4484491fb7 feat: Create ArchiveResult after finishing an extractor process 2020-11-04 11:22:55 -05:00
Cristian
ac0ec160d1 lint: Fix warnings in master branch 2020-11-02 08:51:48 -05:00
Nick Sweeting
ac9e0e356d config fixes 2020-10-31 07:57:11 -04:00
Nick Sweeting
18355dc2c6 clean up config loading in settings and config file layout 2020-10-31 03:08:03 -04:00
Cristian
e7e33ea7a5 tests: Add tests for several different ways to extract the title 2020-10-30 08:04:26 -05:00
Nick Sweeting
f727ece7b3 add regex fallback back to title parser 2020-10-30 04:57:31 -04:00
Nick Sweeting
79bef1384e
Merge pull request #493 from ttimasdf/feat-ogtitle
Feature: add og:title metadata as alternative title
2020-10-30 04:51:14 -04:00
Cristian
c12fe0e3d7 feat: Use CURL_ARGS on title extractor 2020-10-22 08:46:16 -05:00
Cristian
563d0f94ec feat: Use CURL_ARGS in favicon extractor 2020-10-22 08:46:16 -05:00
Cristian
2e1cdca789 feat: Use CURL_ARGS on header extractor 2020-10-22 08:46:16 -05:00
Cristian
972d57bd08 feat: Add CURL_ARGS to control curl arguments 2020-10-22 08:46:16 -05:00
Cristian
24e7a74855 feat: Add WGET_ARGS to control wget arguments 2020-10-22 08:46:16 -05:00
Cristian
bc02e0ffe3 feat: Add config for youtubedl (YOUTUBEDL_ARGS) 2020-10-22 08:46:16 -05:00
Angel Rey
ce71747538 replaced os.path in init extractors 2020-10-02 15:46:39 -05:00
Angel Rey
3fb410a604 Replaced os.path in favicon.py 2020-10-02 15:46:39 -05:00
ttimasdf
eda3836dee feat: add og:title metadata as alternative title 2020-09-27 12:54:52 +08:00
Cristian
abde871a3c fix: Wget absolute path generating issues 2020-09-25 08:24:06 -05:00
Cristian
7d3767b882 fix: oneshot command not running extractors 2020-09-24 12:56:16 -05:00
Cristian
62ed11a5ca fix: Improve headers handling 2020-09-24 12:55:51 -05:00
Angel Rey
a40af98ced removed static file check 2020-09-24 12:55:51 -05:00
Angel Rey
dc160daba8 Fixed lint 2020-09-23 11:07:00 -05:00
Angel Rey
7fd7dced9a Added curl params 2020-09-23 11:07:00 -05:00
Angel Rey
852e3c9cff Added headers extractor 2020-09-23 11:07:00 -05:00
Cristian
eb34a6af62 lint: Fix mercury extractor lint issues 2020-09-23 10:35:39 -05:00
Cristian
46b9e3d536 fix: Fix mercury extractor test 2020-09-23 10:34:05 -05:00