Nick Sweeting
0420662174
switch everywhere to use Snapshot.pk and ArchiveResult.pk instead of id
2024-05-13 05:12:12 -07:00
Nick Sweeting
9733b8d04c
remove accidentally commited db
2024-05-13 02:38:02 -07:00
Nick Sweeting
4f9f22e024
create abid_utils with new ABID type for ArchiveBox IDs
2024-05-13 02:37:48 -07:00
Nick Sweeting
f896e5dbeb
switch from monkey patching WebhookModel to using swappable
2024-05-13 02:36:15 -07:00
Nick Sweeting
e97d779cd3
move monkey patches to dedicated file
2024-05-13 02:35:19 -07:00
Nick Sweeting
b5ad134264
dont wait for ipython history saver thread before shell exit
2024-05-12 19:25:55 -07:00
Nick Sweeting
ce833e8ead
automatically create storage directories and symlinks based on ulid
2024-05-12 05:21:58 -07:00
Nick Sweeting
33bc4622a0
add ulid and typeid to Snapshot and ArchiveResult
2024-05-12 04:45:34 -07:00
Nick Sweeting
0529099639
create cache dir and sqlite db
2024-05-12 01:42:34 -07:00
Nick Sweeting
e1a04729b3
wait for bg threads to finish before exiting
2024-05-12 01:42:20 -07:00
Nick Sweeting
0438924491
remove unused template vars
2024-05-11 22:43:26 -07:00
Nick Sweeting
2b4b6e5b3a
select single snapshot row automatically when hotlinking to admin actions
2024-05-11 22:33:39 -07:00
Nick Sweeting
27809f2976
fix python3.10 nested fstring quotes
2024-05-11 22:33:27 -07:00
Nick Sweeting
baf24d2d6a
change wording on 404 snapshot detail page
2024-05-11 22:33:17 -07:00
Nick Sweeting
8841e8b181
add new live django template for snapshot detail page
2024-05-11 22:33:02 -07:00
Nick Sweeting
457c42bf84
load EXTRACTORS dynamically using importlib.import_module
2024-05-11 22:28:59 -07:00
Nick Sweeting
c7f55fc3ba
make CHROME_USER_DATA_DIR error quieter
2024-05-11 22:16:10 -07:00
Nick Sweeting
102e87578c
re-lock packages for python3.10 because we have to support it still
2024-05-11 15:06:19 -07:00
Nick Sweeting
913590ee39
explain weird use of ellipses magic value
2024-05-11 15:02:43 -07:00
Nick Sweeting
b90ba6c909
change header auth to use X-ArchiveBox-API-Key so it doesnt conflict with other auth headers
2024-05-08 20:02:47 -07:00
Nick Sweeting
4c5a3fba8b
more fixes for wget_output_path
2024-05-07 05:38:29 -07:00
Nick Sweeting
f2729c9dc7
revert wget character limit back to windows, ascii is allows MORE not less
2024-05-07 05:24:16 -07:00
Nick Sweeting
9b21ce490e
add workaround logic to catch paths that are too long or contain unprintable characters
2024-05-07 05:03:23 -07:00
Nick Sweeting
f62cb5fb43
change wget to use stricter ascii filepath normalization
2024-05-07 05:03:01 -07:00
Nick Sweeting
f770bba3cf
fix OSError 36 caused by checking for path that is too long to exist
2024-05-07 04:12:07 -07:00
Nick Sweeting
ef856e8051
Merge branch 'dev' into issue1316
2024-05-06 23:14:16 -07:00
Nick Sweeting
27d5d1ddc8
revert queryset intersection back to union for search results
2024-05-06 23:13:52 -07:00
Nick Sweeting
f472705d10
Change phrasing ( #1419 )
2024-05-06 23:11:26 -07:00
Nick Sweeting
3095265880
fix inner quote not escaped
2024-05-06 23:10:59 -07:00
Brandl
8ccd606973
Fix quotation
...
Fixes:
=> ERROR [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt 1.7s
------
> [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt:
1.665 Traceback (most recent call last):
1.665 File "/usr/local/bin/archivebox", line 5, in <module>
1.665 from archivebox.cli import main
1.665 File "/app/archivebox/cli/__init__.py", line 83, in <module>
1.665 SUBCOMMANDS = list_subcommands()
1.665 ^^^^^^^^^^^^^^^^^^
1.665 File "/app/archivebox/cli/__init__.py", line 43, in list_subcommands
1.665 module = import_module('.archivebox_{}'.format(subcommand), __package__)
1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.665 File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
1.665 return _bootstrap._gcd_import(name[level:], package, level)
1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.666 File "/app/archivebox/cli/archivebox_add.py", line 11, in <module>
1.666 from ..main import add
1.666 File "/app/archivebox/main.py", line 233
1.666 f'COMMIT_HASH={COMMIT_HASH[:7] if COMMIT_HASH else 'unknown'}',
1.666 ^^^^^^^
1.666 SyntaxError: f-string: expecting '}'
2024-05-06 21:04:14 +02:00
Nick Sweeting
027c029316
redact passwords, keys, and secret tokens in admin UI
2024-05-06 11:06:42 -07:00
Nick Sweeting
8667ed29f1
improve API webhooks helptext and change app_label to API
2024-05-06 08:11:01 -07:00
Evan Boehs
f998647350
change phrasing
2024-05-06 10:32:36 -04:00
Nick Sweeting
d782bafe2e
fix storages missing stackfiles error
2024-05-06 07:14:01 -07:00
Nick Sweeting
47666ec26b
show webhooks config in django admin
2024-05-06 07:13:54 -07:00
Nick Sweeting
f067451267
fix django timezone.utc removed in 5.0
2024-05-06 07:13:25 -07:00
Nick Sweeting
c7fc9c004f
add django-signal-webhooks
2024-05-06 06:58:03 -07:00
Nick Sweeting
3805a1730d
add 0002 api migration
2024-04-30 21:45:02 -07:00
Nick Sweeting
e20eb52f15
fix COMMIT_HASH missing error
2024-04-30 21:43:22 -07:00
Nick Sweeting
18a5b6e99c
move EXTERNAL_LOCATIONS to data locations in version output
2024-04-25 21:36:43 -07:00
Nick Sweeting
6a6ae7468e
fix lint errors
2024-04-25 21:36:11 -07:00
Nick Sweeting
4a5ad32040
add django-requests-tracker
2024-04-25 18:02:01 -07:00
Nick Sweeting
af669d2f37
rename api files for clarity
2024-04-25 05:55:47 -07:00
Nick Sweeting
75153252dc
big overhaul of REST API, split into auth, core, and cli methods
2024-04-25 03:56:22 -07:00
Nick Sweeting
e5aba0dc2e
fix urljoin bug causing multiple slashes to be merged into one
2024-04-24 19:41:31 -07:00
Nick Sweeting
6cb357e76c
fix fix_url_from_markdown assertion to be valid url
2024-04-24 19:41:11 -07:00
Nick Sweeting
128419f991
expand comment about markdown url trailing paren trimming
2024-04-24 17:50:18 -07:00
Nick Sweeting
beb3932d80
replace uses of URL_REGEX with find_all_urls to handle markdown better
2024-04-24 17:45:45 -07:00
Nick Sweeting
c6d644be29
Merge branch 'dev' into issue1316
2024-04-24 16:24:16 -07:00
Nick Sweeting
98c5e69203
bump lockfiles
2024-04-24 14:38:21 -07:00
Nick Sweeting
8dcfa93ec6
Merge branch 'main' into dev
2024-04-24 14:32:07 -07:00
Nick Sweeting
17f40f3ada
Merge branch 'dev' into fix-URL_REGEX
2024-04-23 19:53:58 -07:00
Nick Sweeting
c6f8a33a63
Update util.py
2024-04-23 19:53:18 -07:00
Nick Sweeting
63fc317229
minor pylint fixes in logging_util
2024-04-23 17:45:36 -07:00
Nick Sweeting
756e159dfe
add new bin/lock_pkgs.sh to generate package lockfiles consistently
2024-04-23 17:45:36 -07:00
longzai
e4dc2701ef
fix URL_REGEX 2
2024-04-11 15:51:55 +08:00
Brandl
5f9aac18f2
api v1
2024-04-10 01:29:24 +02:00
longzai
4ae765ec27
fix the URL_REGEX used in generic_html parsers
...
Signed-off-by: longzai <437172242@qq.com>
2024-04-08 04:53:05 +08:00
Nick Sweeting
e48159b8a0
cleanup docker-compose by storing crontabs in data dir
2024-03-26 15:24:05 -07:00
Nick Sweeting
ac73fb5129
merge fixes
2024-03-26 15:22:40 -07:00
Nick Sweeting
b4c3aa5097
Merge branch 'main' into dev
2024-03-26 15:01:36 -07:00
Nick Sweeting
a4453b6f87
fix PERSONAS PERSONAS_DIR typo
2024-03-26 14:19:25 -07:00
jim winstead
8b1b01e508
Update to Django 4.2.x, now in LTS until April 2026
2024-03-25 17:46:01 -07:00
Nick Sweeting
c0b5dbcecb
create new data/personas dir to hold cookies and chrome profiles
2024-03-18 14:41:39 -07:00
Nick Sweeting
c5bb99dce1
explicitly use Default profile inside user data dir
2024-03-18 14:40:40 -07:00
Nick Sweeting
1fc5d7c5c8
add USER_AGENT config option to set all USER_AGENTs at once
2024-03-18 14:39:09 -07:00
jim winstead
5478d13d52
Add generic_jsonl parser
...
Resolves #1369
2024-03-14 15:42:29 -07:00
Nick Sweeting
ca2c484a8e
Add _EXTRA_ARGS for various extractors ( #1360 )
2024-03-14 01:55:09 -07:00
Nick Sweeting
48f4b12ae2
Use COOKIES_FILE to fetch page titles ( #1364 )
2024-03-14 01:52:44 -07:00
Nick Sweeting
099f7d00fe
Use feedparser for RSS parsing ( #1362 )
...
Fixes #1171
Fixes #870 (probably, would need to test against a Wallabag Atom file to
Fixes #135
Fixes #123
Fixes #106
2024-03-14 01:51:45 -07:00
Nick Sweeting
3512dc7e60
Disable searching for existing chrome user profiles by default
2024-03-14 00:58:45 -07:00
Ben Muthalaly
f4deb97f59
Add ARGS and EXTRA_ARGS for Mercury extractor
2024-03-05 21:15:38 -06:00
Ben Muthalaly
d8cf09c21e
Remove unnecessary variable length args for dedupe
2024-03-05 21:13:45 -06:00
Ben Muthalaly
5082d61613
Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file
2024-03-05 02:03:03 -06:00
Ben Muthalaly
4686da91e6
Fix cookies being set incorrectly
2024-03-05 01:48:35 -06:00
Naomi Phillips
a729480b75
Add COOKIES_FILE support for singlefile extractor
2024-03-03 02:32:46 -05:00
Ben Muthalaly
d74ddd42ae
Flip dedupe precedence order
2024-03-01 14:50:32 -06:00
jim winstead
9f462a87a8
Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers
...
The feedparser packages has 20 years of history and is very good at parsing
RSS and Atom, so use that instead of ad-hoc regex and XML parsing.
The medium_rss and shaarli_rss parsers weren't touched because they are
probably unnecessary. (The special parse for pinboard is just needing because
of how tags work.)
Doesn't include tests because I haven't figured out how to run them in the
docker development setup.
Fixes #1171
2024-03-01 11:25:45 -08:00
Nick Sweeting
a577d1ed23
Merge branch 'dev' into title-cookies-file
2024-02-29 21:29:36 -08:00
Nick Sweeting
fe11e1c2f4
check if COOKIE_FILE is file
2024-02-28 18:19:44 -08:00
jim winstead
ccabda4c7d
Handle list of tags in JSON, and be more clever about comma vs. space
2024-02-28 17:38:49 -08:00
jim winstead
178e676e0f
Fix JSON parser by not always mangling the input
...
Rather than by assuming the JSON file we are parsing has junk at the beginning
(which maybe only used to happen?), try parsing it as-is first, and then fall
back to trying again after skipping the first line
Fixes #1347
2024-02-27 14:48:19 -08:00
Ben Muthalaly
68326a60ee
Add cookies file to http request in download_url
2024-02-27 15:30:31 -06:00
Ben Muthalaly
4d9c5a7b4b
Add CHROME_EXTRA_ARGS
...
Also fix `YOUTUBEDL_EXTRA_ARGS`.
2024-02-23 18:40:03 -06:00
Ben Muthalaly
ab8f395e0a
Add YOUTUBEDL_EXTRA_ARGS
2024-02-23 15:40:31 -06:00
Ben Muthalaly
4e69d2c9e1
Add EXTRA_*_ARGS for wget, curl, and singlefile
2024-02-22 23:04:11 -06:00
Nick Sweeting
8b9bc3dec8
minor fixes
2024-02-22 04:50:22 -08:00
Nick Sweeting
6a4e568d1b
new archivebox update speed improvements
2024-02-22 04:50:22 -08:00
Nick Sweeting
903c72fa88
fix typing errors
2024-02-12 21:28:52 -08:00
Nick Sweeting
91c4641199
skip dir size calculation when path is too long
2024-02-12 21:26:34 -08:00
Nick Sweeting
19aefc85e6
fix get_system_user failing on uid 999 in k3s
2024-02-08 18:58:12 -08:00
Nick Sweeting
a4bd441077
Update config.py fix trim-filenames
2024-01-31 01:59:43 -08:00
Nick Sweeting
e8772513ca
Update archivebox/core/auth_ldap.py
...
add log line
2024-01-24 11:00:42 -08:00
Vladimir D
914df75ee7
Merge branch 'dev' into feature/is_staff-is_superuser-flags-defaults
2024-01-24 22:25:27 +04:00
Vladimir D
95580ee743
populate is_staff and is_superuser flags at LDAP authentication
2024-01-24 22:18:02 +04:00
Nick Sweeting
3b36928bdc
add yt-dlp restrict-filenames option to make filenames shorter
2024-01-23 14:20:59 -08:00
Nick Sweeting
93781c58ce
add more git clone domains
2024-01-23 13:51:56 -08:00
Nick Sweeting
c7cab7cadc
Show the upgrade notification only in specific views ( #1314 )
2024-01-23 10:29:14 -08:00
Neel Suthar
279883d6bb
Text Search and Filters don't work at the same time in the web UI #1316
...
Making sure to return distinct results. Changing set operation to '&' to show the matching results from filters AND search term
2024-01-21 17:34:22 -06:00
Neel Suthar
aca0512926
Fix for Bug: Search sometimes shows the same snapshot twice #1193
...
Making sure the search results are unique
2024-01-19 17:03:51 -06:00
