Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-10 06:34:16 +00:00
Code Issues Projects Releases Packages Wiki Activity
3982 commits 6 branches 77 tags 18 MiB
Commit graph

87 commits

Author SHA1 Message Date
jim winstead
5478d13d52 Add generic_jsonl parser 
Resolves #1369
 2024-03-14 15:42:29 -07:00
Nick Sweeting
099f7d00fe
Use feedparser for RSS parsing (#1362) 
Fixes #1171
Fixes #870 (probably, would need to test against a Wallabag Atom file to
Fixes #135
Fixes #123
Fixes #106
 2024-03-14 01:51:45 -07:00
jim winstead
741ff5f1a8 Make it a little easier to run specific tests 
Changes ./bin/test.sh to pass command line options to pytest, and default to
only running tests in the tests/ directory instead of everywhere excluding
a few directories which is more error-prone.

Also keeps the mock_server used in testing quiet so access log entries don't
appear on stdout.
 2024-03-01 12:43:53 -08:00
jim winstead
0f402df42f Merge with latest dev 2024-03-01 12:05:43 -08:00
jim winstead
e7119adb0b Add tests for generic_rss and pinboard_rss parsers 2024-03-01 11:27:59 -08:00
jim winstead
1f828d9441 Add tests for generic_rss and pinboard_rss parsers 2024-03-01 11:22:28 -08:00
jim winstead
ccabda4c7d Handle list of tags in JSON, and be more clever about comma vs. space 2024-02-28 17:38:49 -08:00
jim winstead
178e676e0f Fix JSON parser by not always mangling the input 
Rather than by assuming the JSON file we are parsing has junk at the beginning
(which maybe only used to happen?), try parsing it as-is first, and then fall
back to trying again after skipping the first line

Fixes #1347
 2024-02-27 14:48:19 -08:00
Nick Sweeting
a680724367
Merge branch 'dev' into search_index_extract_html_text 2023-10-27 23:09:28 -07:00
Ross Williams
310b4d1242 Add htmltotext extractor 
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
 2023-10-23 21:42:32 -04:00
Ross Williams
b44f7e68b1 Add URL-specific method allow/deny lists 
Allows enabling only allow-listed extractors or disabling specific
deny-listed extractors for a regular expression matched against an added
site's URL.
 2023-08-02 09:36:40 -04:00
Sascha Ißbrücker
40c122515a fix: make oneshot command return successful exist code 2023-05-29 10:01:27 +02:00
Nick Sweeting
9f1470cf03 fix output permissions tests 2021-05-31 20:57:46 -04:00
Nick Sweeting
eef9adbfcb fix select invalid test 2021-04-03 15:50:48 -04:00
Nick Sweeting
354b4627ed fix tests 2021-03-30 23:39:15 -04:00
Nick Sweeting
bd6d9c165b enforce utf8 on literally all file operations because windows sucks 2021-03-27 01:16:29 -04:00
Nick Sweeting
33df9c1ebe fix after and before in remove tests 2021-02-18 06:21:44 -05:00
Nick Sweeting
4f5bb3776c fix sql err 2021-02-18 05:51:53 -05:00
Nick Sweeting
46a4197514 fix tests 2021-02-18 04:26:56 -05:00
Cristian
e82161a768 refactor: Remove setup_django from search 2020-12-11 16:43:48 -05:00
Nick Sweeting
e03d17c208 test extract flag on oneshot 2020-12-11 16:49:18 +02:00
Cristian
f6c73f9aeb fix: Issue with oneshot command 2020-12-08 18:42:25 -05:00
Nick Sweeting
1b22f8eeef
Merge pull request #515 from cdvv7788/POC-setup-django-on-init 2020-11-27 23:56:37 -05:00
Nick Sweeting
efe3027797
Merge branch 'master' into archive-result 2020-11-27 23:18:11 -05:00
Nick Sweeting
0e2ccbc10d update urls to new repo path 2020-11-23 02:06:46 -05:00
Nick Sweeting
fdd4effc92
Merge pull request #535 from cdvv7788/extractors-flag 2020-11-13 14:53:17 -05:00
JDC
b1dbfcb73f Add test remove tag filter 2020-11-13 14:17:12 -05:00
Cristian
44eede96e5 feat: Add extract flag to add command 2020-11-13 09:24:34 -05:00
Cristian
33182fd53c fix: Add missing assignation 2020-11-04 15:07:45 -05:00
Cristian
d064a3eeff fix: Handle case when update tries to re-add a link that is not in the sql index 2020-11-04 15:02:54 -05:00
Cristian
e7e33ea7a5 tests: Add tests for several different ways to extract the title 2020-10-30 08:04:26 -05:00
Cristian
f6ce1de882 fix: archivebox version was being called as root 2020-10-27 09:15:14 -05:00
Cristian
a6bee5f111 feat: Move setup_django to an inner module 2020-10-26 08:02:04 -05:00
Cristian
e1d0b8bce7 feat: Initialize django at the beginning 2020-10-26 07:45:21 -05:00
Cristian
ae1484b8bf feat: Remove index.json and index.html generation from the regular process 2020-10-23 06:45:56 -05:00
Cristian Vargas
a850b4a9d9
Merge branch 'master' into tags 2020-10-20 08:23:25 -05:00
Cristian
62c78e1d10 refactor: Remove django-taggit and replace it with a local tags setup 2020-10-12 13:47:03 -05:00
Angel Rey
73418836f8 Replaced os.path in server.py 2020-10-02 15:46:39 -05:00
Angel Rey
62c9028212 Improved tags 2020-09-24 15:34:23 -05:00
Cristian
0158efb1d0 test: Improve oneshot test 2020-09-24 12:56:16 -05:00
Cristian
62ed11a5ca fix: Improve headers handling 2020-09-24 12:55:51 -05:00
Angel Rey
ee6caca3ca Added more asserts 2020-09-23 11:07:00 -05:00
Angel Rey
1cce786d6d Added test headers extractor 2020-09-23 11:07:00 -05:00
Cristian
46b9e3d536 fix: Fix mercury extractor test 2020-09-23 10:34:05 -05:00
ttimasdf
e3329be291 tests: add test for mercury-parser 2020-09-22 18:44:12 -05:00
Cristian
fa622d3e14 refactor: Replace --index with --with-headers in the list command to make it more explicit. Change it so it affects the csv output too. 2020-09-15 08:05:46 -05:00
Cristian
2aa8d69b72 fix: Save history in main index (to mimic previous behaviour) 2020-09-15 08:05:46 -05:00
Cristian
7e9d195d13 feat: Update list command to sort using sqlite 2020-09-15 08:05:46 -05:00
Cristian
f55153eab3 feat: Update update command to work with querysets 2020-09-15 08:05:46 -05:00
Cristian
dafa1dd63c tests: Add tests for before and after flags in remove command 2020-09-15 08:05:46 -05:00