Nick Sweeting
406f57031a
add API support for obj.pk .uuid .abid
2024-05-13 08:52:02 -07:00
Nick Sweeting
a4cc10d7f8
add migrations for third round of field changes
2024-05-13 07:50:22 -07:00
Nick Sweeting
241a7c6ab2
add created, modified, updated, created_by and update django admin
2024-05-13 07:50:07 -07:00
Nick Sweeting
1ba8215072
only use domain part of uri for hash
2024-05-13 07:49:36 -07:00
Nick Sweeting
206e7e74b3
add migrations to create and populate ABIDField and UUIDField values
2024-05-13 05:13:42 -07:00
Nick Sweeting
0420662174
switch everywhere to use Snapshot.pk and ArchiveResult.pk instead of id
2024-05-13 05:12:12 -07:00
Nick Sweeting
9733b8d04c
remove accidentally commited db
2024-05-13 02:38:02 -07:00
Nick Sweeting
4f9f22e024
create abid_utils with new ABID type for ArchiveBox IDs
2024-05-13 02:37:48 -07:00
Nick Sweeting
f896e5dbeb
switch from monkey patching WebhookModel to using swappable
2024-05-13 02:36:15 -07:00
Nick Sweeting
e97d779cd3
move monkey patches to dedicated file
2024-05-13 02:35:19 -07:00
Nick Sweeting
b5ad134264
dont wait for ipython history saver thread before shell exit
2024-05-12 19:25:55 -07:00
Nick Sweeting
ce833e8ead
automatically create storage directories and symlinks based on ulid
2024-05-12 05:21:58 -07:00
Nick Sweeting
33bc4622a0
add ulid and typeid to Snapshot and ArchiveResult
2024-05-12 04:45:34 -07:00
Nick Sweeting
0529099639
create cache dir and sqlite db
2024-05-12 01:42:34 -07:00
Nick Sweeting
e1a04729b3
wait for bg threads to finish before exiting
2024-05-12 01:42:20 -07:00
Nick Sweeting
0438924491
remove unused template vars
2024-05-11 22:43:26 -07:00
Nick Sweeting
2b4b6e5b3a
select single snapshot row automatically when hotlinking to admin actions
2024-05-11 22:33:39 -07:00
Nick Sweeting
27809f2976
fix python3.10 nested fstring quotes
2024-05-11 22:33:27 -07:00
Nick Sweeting
baf24d2d6a
change wording on 404 snapshot detail page
2024-05-11 22:33:17 -07:00
Nick Sweeting
8841e8b181
add new live django template for snapshot detail page
2024-05-11 22:33:02 -07:00
Nick Sweeting
457c42bf84
load EXTRACTORS dynamically using importlib.import_module
2024-05-11 22:28:59 -07:00
Nick Sweeting
c7f55fc3ba
make CHROME_USER_DATA_DIR error quieter
2024-05-11 22:16:10 -07:00
Nick Sweeting
102e87578c
re-lock packages for python3.10 because we have to support it still
2024-05-11 15:06:19 -07:00
Nick Sweeting
913590ee39
explain weird use of ellipses magic value
2024-05-11 15:02:43 -07:00
Nick Sweeting
b90ba6c909
change header auth to use X-ArchiveBox-API-Key so it doesnt conflict with other auth headers
2024-05-08 20:02:47 -07:00
Nick Sweeting
4c5a3fba8b
more fixes for wget_output_path
2024-05-07 05:38:29 -07:00
Nick Sweeting
f2729c9dc7
revert wget character limit back to windows, ascii is allows MORE not less
2024-05-07 05:24:16 -07:00
Nick Sweeting
9b21ce490e
add workaround logic to catch paths that are too long or contain unprintable characters
2024-05-07 05:03:23 -07:00
Nick Sweeting
f62cb5fb43
change wget to use stricter ascii filepath normalization
2024-05-07 05:03:01 -07:00
Nick Sweeting
f770bba3cf
fix OSError 36 caused by checking for path that is too long to exist
2024-05-07 04:12:07 -07:00
Nick Sweeting
ef856e8051
Merge branch 'dev' into issue1316
2024-05-06 23:14:16 -07:00
Nick Sweeting
27d5d1ddc8
revert queryset intersection back to union for search results
2024-05-06 23:13:52 -07:00
Nick Sweeting
f472705d10
Change phrasing ( #1419 )
2024-05-06 23:11:26 -07:00
Nick Sweeting
3095265880
fix inner quote not escaped
2024-05-06 23:10:59 -07:00
Brandl
8ccd606973
Fix quotation
...
Fixes:
=> ERROR [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt 1.7s
------
> [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt:
1.665 Traceback (most recent call last):
1.665 File "/usr/local/bin/archivebox", line 5, in <module>
1.665 from archivebox.cli import main
1.665 File "/app/archivebox/cli/__init__.py", line 83, in <module>
1.665 SUBCOMMANDS = list_subcommands()
1.665 ^^^^^^^^^^^^^^^^^^
1.665 File "/app/archivebox/cli/__init__.py", line 43, in list_subcommands
1.665 module = import_module('.archivebox_{}'.format(subcommand), __package__)
1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.665 File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
1.665 return _bootstrap._gcd_import(name[level:], package, level)
1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.666 File "/app/archivebox/cli/archivebox_add.py", line 11, in <module>
1.666 from ..main import add
1.666 File "/app/archivebox/main.py", line 233
1.666 f'COMMIT_HASH={COMMIT_HASH[:7] if COMMIT_HASH else 'unknown'}',
1.666 ^^^^^^^
1.666 SyntaxError: f-string: expecting '}'
2024-05-06 21:04:14 +02:00
Nick Sweeting
027c029316
redact passwords, keys, and secret tokens in admin UI
2024-05-06 11:06:42 -07:00
Nick Sweeting
8667ed29f1
improve API webhooks helptext and change app_label to API
2024-05-06 08:11:01 -07:00
Evan Boehs
f998647350
change phrasing
2024-05-06 10:32:36 -04:00
Nick Sweeting
d782bafe2e
fix storages missing stackfiles error
2024-05-06 07:14:01 -07:00
Nick Sweeting
47666ec26b
show webhooks config in django admin
2024-05-06 07:13:54 -07:00
Nick Sweeting
f067451267
fix django timezone.utc removed in 5.0
2024-05-06 07:13:25 -07:00
Nick Sweeting
c7fc9c004f
add django-signal-webhooks
2024-05-06 06:58:03 -07:00
Nick Sweeting
3805a1730d
add 0002 api migration
2024-04-30 21:45:02 -07:00
Nick Sweeting
e20eb52f15
fix COMMIT_HASH missing error
2024-04-30 21:43:22 -07:00
Nick Sweeting
18a5b6e99c
move EXTERNAL_LOCATIONS to data locations in version output
2024-04-25 21:36:43 -07:00
Nick Sweeting
6a6ae7468e
fix lint errors
2024-04-25 21:36:11 -07:00
Nick Sweeting
4a5ad32040
add django-requests-tracker
2024-04-25 18:02:01 -07:00
Nick Sweeting
af669d2f37
rename api files for clarity
2024-04-25 05:55:47 -07:00
Nick Sweeting
75153252dc
big overhaul of REST API, split into auth, core, and cli methods
2024-04-25 03:56:22 -07:00
Nick Sweeting
e5aba0dc2e
fix urljoin bug causing multiple slashes to be merged into one
2024-04-24 19:41:31 -07:00
Nick Sweeting
6cb357e76c
fix fix_url_from_markdown assertion to be valid url
2024-04-24 19:41:11 -07:00
Nick Sweeting
128419f991
expand comment about markdown url trailing paren trimming
2024-04-24 17:50:18 -07:00
Nick Sweeting
beb3932d80
replace uses of URL_REGEX with find_all_urls to handle markdown better
2024-04-24 17:45:45 -07:00
Nick Sweeting
c6d644be29
Merge branch 'dev' into issue1316
2024-04-24 16:24:16 -07:00
Nick Sweeting
98c5e69203
bump lockfiles
2024-04-24 14:38:21 -07:00
Nick Sweeting
8dcfa93ec6
Merge branch 'main' into dev
2024-04-24 14:32:07 -07:00
Nick Sweeting
17f40f3ada
Merge branch 'dev' into fix-URL_REGEX
2024-04-23 19:53:58 -07:00
Nick Sweeting
c6f8a33a63
Update util.py
2024-04-23 19:53:18 -07:00
Nick Sweeting
63fc317229
minor pylint fixes in logging_util
2024-04-23 17:45:36 -07:00
Nick Sweeting
756e159dfe
add new bin/lock_pkgs.sh to generate package lockfiles consistently
2024-04-23 17:45:36 -07:00
longzai
e4dc2701ef
fix URL_REGEX 2
2024-04-11 15:51:55 +08:00
Brandl
5f9aac18f2
api v1
2024-04-10 01:29:24 +02:00
longzai
4ae765ec27
fix the URL_REGEX used in generic_html parsers
...
Signed-off-by: longzai <437172242@qq.com>
2024-04-08 04:53:05 +08:00
Nick Sweeting
e48159b8a0
cleanup docker-compose by storing crontabs in data dir
2024-03-26 15:24:05 -07:00
Nick Sweeting
ac73fb5129
merge fixes
2024-03-26 15:22:40 -07:00
Nick Sweeting
b4c3aa5097
Merge branch 'main' into dev
2024-03-26 15:01:36 -07:00
Nick Sweeting
a4453b6f87
fix PERSONAS PERSONAS_DIR typo
2024-03-26 14:19:25 -07:00
jim winstead
8b1b01e508
Update to Django 4.2.x, now in LTS until April 2026
2024-03-25 17:46:01 -07:00
Nick Sweeting
c0b5dbcecb
create new data/personas dir to hold cookies and chrome profiles
2024-03-18 14:41:39 -07:00
Nick Sweeting
c5bb99dce1
explicitly use Default profile inside user data dir
2024-03-18 14:40:40 -07:00
Nick Sweeting
1fc5d7c5c8
add USER_AGENT config option to set all USER_AGENTs at once
2024-03-18 14:39:09 -07:00
jim winstead
5478d13d52
Add generic_jsonl parser
...
Resolves #1369
2024-03-14 15:42:29 -07:00
Nick Sweeting
ca2c484a8e
Add _EXTRA_ARGS for various extractors ( #1360 )
2024-03-14 01:55:09 -07:00
Nick Sweeting
48f4b12ae2
Use COOKIES_FILE to fetch page titles ( #1364 )
2024-03-14 01:52:44 -07:00
Nick Sweeting
099f7d00fe
Use feedparser for RSS parsing ( #1362 )
...
Fixes #1171
Fixes #870 (probably, would need to test against a Wallabag Atom file to
Fixes #135
Fixes #123
Fixes #106
2024-03-14 01:51:45 -07:00
Nick Sweeting
3512dc7e60
Disable searching for existing chrome user profiles by default
2024-03-14 00:58:45 -07:00
Ben Muthalaly
f4deb97f59
Add ARGS and EXTRA_ARGS for Mercury extractor
2024-03-05 21:15:38 -06:00
Ben Muthalaly
d8cf09c21e
Remove unnecessary variable length args for dedupe
2024-03-05 21:13:45 -06:00
Ben Muthalaly
5082d61613
Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file
2024-03-05 02:03:03 -06:00
Ben Muthalaly
4686da91e6
Fix cookies being set incorrectly
2024-03-05 01:48:35 -06:00
Naomi Phillips
a729480b75
Add COOKIES_FILE support for singlefile extractor
2024-03-03 02:32:46 -05:00
Ben Muthalaly
d74ddd42ae
Flip dedupe precedence order
2024-03-01 14:50:32 -06:00
jim winstead
9f462a87a8
Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers
...
The feedparser packages has 20 years of history and is very good at parsing
RSS and Atom, so use that instead of ad-hoc regex and XML parsing.
The medium_rss and shaarli_rss parsers weren't touched because they are
probably unnecessary. (The special parse for pinboard is just needing because
of how tags work.)
Doesn't include tests because I haven't figured out how to run them in the
docker development setup.
Fixes #1171
2024-03-01 11:25:45 -08:00
Nick Sweeting
a577d1ed23
Merge branch 'dev' into title-cookies-file
2024-02-29 21:29:36 -08:00
Nick Sweeting
fe11e1c2f4
check if COOKIE_FILE is file
2024-02-28 18:19:44 -08:00
jim winstead
ccabda4c7d
Handle list of tags in JSON, and be more clever about comma vs. space
2024-02-28 17:38:49 -08:00
jim winstead
178e676e0f
Fix JSON parser by not always mangling the input
...
Rather than by assuming the JSON file we are parsing has junk at the beginning
(which maybe only used to happen?), try parsing it as-is first, and then fall
back to trying again after skipping the first line
Fixes #1347
2024-02-27 14:48:19 -08:00
Ben Muthalaly
68326a60ee
Add cookies file to http request in download_url
2024-02-27 15:30:31 -06:00
Ben Muthalaly
4d9c5a7b4b
Add CHROME_EXTRA_ARGS
...
Also fix `YOUTUBEDL_EXTRA_ARGS`.
2024-02-23 18:40:03 -06:00
Ben Muthalaly
ab8f395e0a
Add YOUTUBEDL_EXTRA_ARGS
2024-02-23 15:40:31 -06:00
Ben Muthalaly
4e69d2c9e1
Add EXTRA_*_ARGS for wget, curl, and singlefile
2024-02-22 23:04:11 -06:00
Nick Sweeting
8b9bc3dec8
minor fixes
2024-02-22 04:50:22 -08:00
Nick Sweeting
6a4e568d1b
new archivebox update speed improvements
2024-02-22 04:50:22 -08:00
Nick Sweeting
903c72fa88
fix typing errors
2024-02-12 21:28:52 -08:00
Nick Sweeting
91c4641199
skip dir size calculation when path is too long
2024-02-12 21:26:34 -08:00
Nick Sweeting
19aefc85e6
fix get_system_user failing on uid 999 in k3s
2024-02-08 18:58:12 -08:00
Nick Sweeting
a4bd441077
Update config.py fix trim-filenames
2024-01-31 01:59:43 -08:00
Nick Sweeting
e8772513ca
Update archivebox/core/auth_ldap.py
...
add log line
2024-01-24 11:00:42 -08:00
Vladimir D
914df75ee7
Merge branch 'dev' into feature/is_staff-is_superuser-flags-defaults
2024-01-24 22:25:27 +04:00
Vladimir D
95580ee743
populate is_staff and is_superuser flags at LDAP authentication
2024-01-24 22:18:02 +04:00
Nick Sweeting
3b36928bdc
add yt-dlp restrict-filenames option to make filenames shorter
2024-01-23 14:20:59 -08:00
Nick Sweeting
93781c58ce
add more git clone domains
2024-01-23 13:51:56 -08:00
Nick Sweeting
c7cab7cadc
Show the upgrade notification only in specific views ( #1314 )
2024-01-23 10:29:14 -08:00
Neel Suthar
279883d6bb
Text Search and Filters don't work at the same time in the web UI #1316
...
Making sure to return distinct results. Changing set operation to '&' to show the matching results from filters AND search term
2024-01-21 17:34:22 -06:00
Neel Suthar
aca0512926
Fix for Bug: Search sometimes shows the same snapshot twice #1193
...
Making sure the search results are unique
2024-01-19 17:03:51 -06:00
Neel Suthar
702b81f9e6
Fixes #1193 Bug: Search sometimes shows the same snapshot twice
...
Making sure user gets the distinct set of search results
2024-01-19 13:49:09 -06:00
Nick Sweeting
d0cd84a2af
Merge branch 'dev' into specific-version-banner
2024-01-19 04:01:32 -08:00
Nick Sweeting
1773146833
include more output file locations when considering whether snapshot.is_archived
2024-01-19 03:47:38 -08:00
Nick Sweeting
0a25495520
add fallback to check wget output dir with port stripped
2024-01-19 03:47:38 -08:00
Nick Sweeting
ef1a697588
fix sorting by Size or by Files to sort by number of archive results
2024-01-19 03:47:38 -08:00
Nick Sweeting
2e77c3964f
persist snapshot index header collapse state
2024-01-19 03:47:38 -08:00
Nick Sweeting
f839f4f16e
Update navigation.html to fix target blank inside django tag typo
2024-01-16 00:37:55 -08:00
Nick Sweeting
8e03755a7a
Make "DOCS" link in navigation bar open in new tab ( #1323 )
2024-01-15 12:36:51 -08:00
Tristan Sherliker
d6a8e12482
Update navigation.html
...
change navigation template so docs link opens in new tab
2024-01-14 16:40:11 +00:00
Nick Sweeting
8c07b7e127
disable automatic chrome selfupdating
2024-01-11 19:51:27 -08:00
Nick Sweeting
9766de1b70
bump version of dev build to 0.7.3
2024-01-11 19:08:13 -08:00
Nick Sweeting
6184f659dc
improve window size chrome cli handling
2024-01-11 19:02:46 -08:00
Ben Muthalaly
5de45dbf30
Show upgrade notification in admin snapshot view
2024-01-08 22:55:30 -06:00
Nick Sweeting
1532186283
Merge branch 'main' into dev
2024-01-04 12:30:46 -08:00
Nick Sweeting
a215b38681
update dompurify version
2024-01-04 11:43:50 -08:00
Nick Sweeting
94b132818d
fix lint errors
2024-01-04 11:41:12 -08:00
Nick Sweeting
c1fd2cfa42
tag URLs immediately once added instead of waiting until archival completes
2024-01-03 20:31:46 -08:00
Nick Sweeting
8ab17f0561
add tag support to schedule command
2024-01-03 20:31:14 -08:00
Nick Sweeting
f775da299a
show command to enter docker container in failed command output hint when IN_DOCKER=True
2024-01-03 20:12:03 -08:00
Nick Sweeting
b9e05b2abe
quote URLs in failure message hint output to make copy-pasting easier
2024-01-03 20:11:30 -08:00
Nick Sweeting
db2984e47b
prefer dom dump to singlefile for generating readability output
2024-01-03 20:11:06 -08:00
Nick Sweeting
78d942ac22
show more detail in readabiliity error messages
2024-01-03 20:09:31 -08:00
Nick Sweeting
c26d9dc1bb
link to exact build commit hash in public index footer
2024-01-03 20:09:04 -08:00
Nick Sweeting
ceabc4e12e
fix CHROME_USER_AGENT not interpolating inline config variables
2024-01-03 20:08:39 -08:00
Nick Sweeting
5b07a1126c
add comment about why DOM is preferred over singlefile for readability parsing
2024-01-03 19:09:24 -08:00
Nick Sweeting
fcdc41a1ab
re-enable autocomplete tag field for now as its breaking action buttons display
2024-01-03 19:09:14 -08:00
Nick Sweeting
b0da386793
include git refs and HEAD file in docker images to allow GIT_SHA of builds to show in version output
2024-01-03 18:59:45 -08:00
Nick Sweeting
4af9beedd1
disable GLOBAL_CONTEXT used by version upgrade banner as not all admin views accept extra_context kwarg
2024-01-03 16:11:27 -08:00
Nick Sweeting
2c54e55697
prefer dom dump to singlefile for generating readability output
2024-01-02 19:50:56 -08:00
Nick Sweeting
8ee2981957
fix check-for-update fetching code
2024-01-02 17:17:35 -08:00
Nick Sweeting
73993d26c0
move SYSTEM_USER call to get_system_user func
2024-01-02 17:11:29 -08:00
Nick Sweeting
6584f300d6
add better comments
2024-01-02 17:10:22 -08:00
Nick Sweeting
bfcff2f78f
make sure user is a superuser before showing upgrade notification
2023-12-19 10:03:04 -08:00
Nick Sweeting
bfebc23f42
Rename VERSION_RELEASES to VERSIONS_AVAILABLE
2023-12-19 10:02:28 -08:00
Nick Sweeting
42dad5e0b3
rename VERSION_RELEASES to VERSIONS_AVAILABLE
2023-12-19 10:01:08 -08:00
Nick Sweeting
168cdac2c4
rename VERSION_RELEASES to VERSIONS_AVAILABLE
2023-12-19 09:58:42 -08:00
Nick Sweeting
480eff52db
Update urls.py
2023-12-19 09:58:15 -08:00
Nick Sweeting
14f10a0461
Merge branch 'dev' into version-banner
2023-12-19 09:57:08 -08:00
Nick Sweeting
487e560e32
Update config.py to use fallback git head method
2023-12-18 19:27:52 -08:00
Nick Sweeting
5805e18380
Update settings.py
2023-12-18 19:25:08 -08:00
Nick Sweeting
7d5a7f80ca
Update admin.py
2023-12-18 19:24:48 -08:00
Nick Sweeting
7d06836443
add WIP comment
2023-12-18 19:24:18 -08:00
Nick Sweeting
89bdda85e9
Merge branch 'dev' into side-fixes
2023-12-18 19:17:31 -08:00
Nick Sweeting
7410fddacf
improve entrypoint error handling and color coding
2023-12-17 21:03:01 -08:00
Nick Sweeting
4a4ed2cad6
fix version string parsing
2023-12-17 17:44:26 -08:00
Nick Sweeting
d5f1411ed1
exit with running as root error when PUID is 0
2023-12-17 16:57:40 -08:00
Nick Sweeting
8a306dbf6f
add BUILD_TIME to archivebox version output
2023-12-17 16:57:02 -08:00
Nick Sweeting
951bba52a0
hide +editable from version string in output because all docker installs are editable
2023-12-17 16:14:39 -08:00
Nick Sweeting
169b427fde
bump min required python version to 3.7.0
2023-12-17 16:05:00 -08:00
Nick Sweeting
d4b0660b8d
add hint to use docker run instead of exec when catching root error fixes #1287
2023-12-17 16:04:02 -08:00
Vladimir D
37bdcbf950
make local auth working along with ldap auth so that local superuser could manage ldap users
2023-12-06 16:16:51 +04:00
Ben Muthalaly
a2b11b696d
Remove version comparison function
2023-12-01 16:55:07 -06:00
Ben Muthalaly
25d7f3ed1c
Parse without regex and change 'let' to 'const'
2023-11-30 19:57:46 -06:00
Ben Muthalaly
4ae35c86f2
Update archivebox/config.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2023-11-30 19:53:20 -06:00
Ben Muthalaly
8616cd758a
Fix typos, formatting, and variable declarations
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2023-11-30 18:06:54 -06:00
Ben Muthalaly
ba4342dadf
Add CLI hints to add and schedule jobs
2023-11-27 22:58:13 -06:00
Ben Muthalaly
a3fd8a8ecd
Move version banner logic from JS to Python
...
Also adds CSS styling to banner.
2023-11-27 22:56:30 -06:00
Ben Muthalaly
7599dbb79d
Add notification about upgrade to admin page
2023-11-23 03:04:31 -06:00
Nick Sweeting
f5c2cfcfe2
dont alert on statics
2023-11-14 02:30:59 -08:00
Nick Sweeting
6761ed368c
wip pre-0.7.1 side fixes
2023-11-14 02:09:58 -08:00
Nick Sweeting
f0033f75d0
config.py lint fixes
2023-11-14 02:07:35 -08:00
Nick Sweeting
0b950b3253
make get_commit_hash more precisce by using exact ref
2023-11-14 02:04:57 -08:00
Nick Sweeting
f6ea21eabf
bump version in package.json
2023-11-14 00:52:29 -08:00
Nick Sweeting
6188dcdcbd
return placeholder version during docs build
2023-11-14 00:38:37 -08:00
Nick Sweeting
d3d05f5f64
add fallback version detection mode
2023-11-14 00:21:09 -08:00
Nick Sweeting
4523d6817f
readme fixes
2023-11-13 23:51:46 -08:00
Nick Sweeting
716f68c34a
bump vendor dependency versions
2023-11-13 22:38:19 -08:00
Nick Sweeting
f5739506f6
Merge pull request #1244 from overhacked/search_index_extract_html_text
2023-11-08 23:34:44 -08:00
Nick Sweeting
7377d9ebc3
bump version to 0.7.1 and fix version autodetection code
2023-11-03 20:07:39 -07:00
Nick Sweeting
db733151f7
bump user agent browser version
2023-10-31 19:16:15 -07:00
Nick Sweeting
166573d417
autodetect qemu and warn about broken chrome in emulated containers
2023-10-31 00:23:19 -07:00
Nick Sweeting
62e077a5bc
Merge pull request #1241 from overhacked/fts5-search
2023-10-30 18:49:52 -07:00
Nick Sweeting
851f59b4ba
Merge pull request #1254 from c01o/dev
2023-10-30 18:44:55 -07:00
mAAdhaTTah
18e89bee3d
Fix readwise token
...
Accidentally shipped this hard-coded. Token is long-size revoked,
but obviously hard to use this way!
2023-10-29 17:27:04 -04:00
Nick Sweeting
a680724367
Merge branch 'dev' into search_index_extract_html_text
2023-10-27 23:09:28 -07:00
c01o
8c25a81009
Remove playwright from armv7
2023-10-27 05:59:25 +09:00
Ross Williams
310b4d1242
Add htmltotext extractor
...
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
2023-10-23 21:42:32 -04:00
Nick Sweeting
63ad43f46c
Merge branch 'dev' into method_allow_deny
2023-10-20 04:25:44 -07:00
Nick Sweeting
53355be46a
Merge branch 'dev' into fts5-search
2023-10-20 04:23:50 -07:00
Nick Sweeting
82d8662c74
add more readability error output
2023-10-20 04:14:28 -07:00
Nick Sweeting
76f9b91ed3
dockerfile fixes
2023-10-20 04:08:38 -07:00
Nick Sweeting
86c662421b
show FS_USER in version output debug string
2023-10-20 02:47:23 -07:00
Nick Sweeting
7910a5b34b
fix pretty_path printing logic
2023-10-20 02:47:03 -07:00
Nick Sweeting
d4ca2d1154
update mercury bin path to postlight parser
2023-10-20 02:46:41 -07:00
Nick Sweeting
63c276a93d
redirect add page back to snapshots list automatically
2023-10-19 18:24:19 -07:00
Nick Sweeting
22bcffe5ee
remove accidental duplicate template code
2023-10-19 18:24:05 -07:00
Nick Sweeting
85f8583d62
catch ldap configuration and packaging errors and make them non fatal
2023-10-19 18:05:59 -07:00
Nick Sweeting
d286dca925
better to log before doing a thing than after
2023-10-18 11:47:55 -07:00
Ben Muthalaly
9e6a87114b
Fix formatting, logging, logic issues
2023-10-18 12:07:36 -05:00
Ben Muthalaly
521ea70e0c
Add check for existing user, change varable names
2023-10-18 03:07:54 -05:00
Ross Williams
9b85f35b63
sqlite search: check SQLite version when indexing
...
If creating the FTS5 tables fails due to a known version
incompatiblity, report the required version to the user.
2023-10-16 14:50:48 -04:00
Ross Williams
1e604a1352
sqlite search: clean up errors and type-checking
...
Clean up error handling, and report a better error message
on search and flush if FTS5 tables haven't yet been created.
Add some mypy comments to clean up type-checking errors.
2023-10-16 14:31:52 -04:00
Ross Williams
adb9f0ecc9
sqlite search: Rename connection to get_connection
...
`connection` could cause confusion with `django.db.connection` and
`get_connection` is a better callable name.
2023-10-16 13:23:26 -04:00
Ross Williams
e0f8eeeaa7
Improve search.backends.sqlite retry logic
...
Retry with table creation should fail if it is attempted for a second
time.
2023-10-16 13:23:26 -04:00
Ross Williams
8fe5faf4d0
Introduce SQLite FTS5-powered search backend
...
Use SQLite's FTS5 extension to power full-text search without any
additional dependencies. FTS5 was introduced in SQLite 3.9.0,
[released][1] in 2015 so should be available on most SQLite
installations at this point in time.
[1]: https://www.sqlite.org/changes.html#version_3_9_0
2023-10-16 13:23:26 -04:00
Ross Williams
c53ec45a29
WIP: add sqlite search backend boilerplate
2023-10-16 13:23:26 -04:00
Ross Williams
6555719489
Add space after tags when extracting text
...
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.
Example:
`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
2023-10-16 09:59:08 -04:00
Ben Muthalaly
44a94157be
Remove logging of configured username and password
2023-10-15 23:36:47 -05:00
Ben Muthalaly
11d473e536
Add config options to add admin user on first run
2023-10-14 00:38:04 -05:00
Ross Williams
d8aa84ac98
Make extracting text for indexing optional
...
Add a configuration option to enable/disable HTML text extraction
for indexing
2023-10-12 13:14:39 -04:00
Ross Williams
b6a20c962a
Extract text from singlefile.html when indexing
...
singlefile.html contains a lot of large strings in the form of `data:`
URLs, which can be unnecessarily stored in full-text indices. Also,
large chunks of JavaScript shouldn't be indexed, either, as they pollute
search results for searches about JS functions, etc.
This commit takes a blanket approach of parsing singlefile.html as it is
read and only outputting text and selected textual attributes (like
`alt`) for indexing.
2023-10-12 13:06:35 -04:00
Ben Muthalaly
77917e9b55
Fix HTML title parsing bugs.
...
This slightly modifies the HTML_TITLE_REGEX to fix two parsing errors.
The first occurred when title tags were empty (e.g. "<title></title>")
which was parsed as "</title". The second occurred when titles were a
single character (e.g. "<title>A</title>") which was not matched by the
regex, and so would fall back to link.base_url.
Now when tags are empty, it falls back to link.base_url, and single
character titles are parsed correctly.
The way the regex works now is still a bit wonky for some edge cases.
I couldn't find any cases of incorrect behavior, but it still might be
worth reworking more completely for robustness.
2023-10-09 02:00:01 -05:00
Nick Sweeting
5c1a14e4f2
ignore errors while getting system user name
2023-09-14 03:39:44 -07:00
Nick Sweeting
ffe2968e4f
improve some comments
2023-09-14 02:41:27 -07:00
Nick Sweeting
f809efce4d
Merge pull request #996 from barthalion/dev
2023-09-03 21:40:49 -07:00
Nick Sweeting
aaca74f6a8
only start parsing json after the first open brace
2023-09-03 21:40:12 -07:00
Nick Sweeting
cd9f228b2f
Merge pull request #1214 from DanielBatteryStapler/DanielBatteryStapler-patch-1
2023-09-03 21:25:12 -07:00
Nick Sweeting
16d278fbdb
Merge pull request #1168 from mAAdhaTTah/add-readwise-reader
2023-09-03 21:24:49 -07:00
Nick Sweeting
110a22ee32
Merge branch 'dev' into DanielBatteryStapler-patch-1
2023-08-31 15:20:46 -07:00
Nick Sweeting
73a5f74d38
update default YOUTUBEDL_ARGS to fix subs and filesize
2023-08-31 15:17:45 -07:00
Nick Sweeting
86366d5640
Update logging_util.py to fix generator subscripting error
2023-08-31 15:12:43 -07:00
spresse1
603ce7ec10
After a timeout, chrome will leave behind a SingletonLock, which prevents future instances of chrome from starting. When an extractor fails due to a timeout, remove this file.
2023-08-28 17:27:03 +02:00
root
23f086aa40
add LDAP support
2023-08-17 19:51:02 -05:00
DanielBatteryStapler
94dacc49c7
Fix archive_org icon "exists"
2023-08-15 23:49:54 -04:00
Ross Williams
c039ef05b3
Fix hyphen placement in util.URL_REGEX
...
Incorrect hyphen placement in `URL_REGEX` was allowing it to match more
characters than intended. In a regex character class, a literal hyphen
can only appear as the first character in the class, or it will be
interpreted as the delimiter of a range of characters.
The issue fixed here caused the range of characters from `[$-_]`
be treated as valid URL characters, instead of the intended set of three
characters `[-_$]`. The incorrect range interpretation inadvertantly
included most ASCII punctuation, most importantly the angle brackets,
square brackets, and single quote that the expression uses
to mark the end of a match.
This causes the expression to match a URL that has a "hostname" portion
beginning with one of the intended "stop parsing" characters. For
example:
```
https://<b>www</b>.example.com/ # MATCHES but should not
https://[for example] # MATCHES but should not
scheme='https://' # MATCHES, including final quote, but should not
```
Some test cases have been added to the `URL_REGEX` assert in
archivebox.parsers to cover this possibility.
2023-08-08 15:24:16 -04:00
Ross Williams
2076474252
Drop use of TypeAlias to maintain Python 3.9 compat
...
TypeAlias annotation was introduced in Python 3.10, and is not strictly
necessary. Drop use of it to maintain Python 3.9 compatibility.
2023-08-02 10:56:48 -04:00
Ross Williams
b44f7e68b1
Add URL-specific method allow/deny lists
...
Allows enabling only allow-listed extractors or disabling specific
deny-listed extractors for a regular expression matched against an added
site's URL.
2023-08-02 09:36:40 -04:00
Ross Williams
46e80dd509
Rename URL_(WHITE|BLACK)LIST to URL_(ALLOW|DENY)LIST
...
Retain aliases for old configuration files
2023-08-02 09:31:48 -04:00
Nick Sweeting
b773041952
Merge pull request #1199 from overhacked/chrome_version_detection_fix
2023-08-01 10:14:18 -07:00
Ross Williams
d0e65eba7f
More reliably detect Google Chrome version number
...
Previous method was splitting on the first whitespace, and missing the
version number when it appeared as `"Google Chrome 115.0.234.2342"`
instead of, i.e. `"Chromium 115.0.234.8283"`.
This commit changes the version detection to regex search for
whitespace, then one or more digits followed by a period, then at least
one more digit. Only the first sequence of digits is captured. Unless
Chrome radically changes their version numbering, this should capture
the first group of digits after the reported browser name, which would
be the major version.
2023-07-31 15:34:58 -04:00
Ross Williams
9d9872d325
bin_version means to modify, not replace environ
...
the `bin_version` function means to modify the environment,
not replace it entirely. Fixes bugs that occur when it wipes out the
PATH environment variable, such as when running in a virtual
environment.
2023-07-31 11:36:34 -04:00
mAAdhaTTah
181501fd36
Add Readwise Reader API parser
...
Implemented similar to the Pocket API.
2023-07-02 11:20:58 -04:00
Sascha Ißbrücker
7bf4f40da0
just use out_dir
2023-05-29 10:03:49 +02:00
Sascha Ißbrücker
40c122515a
fix: make oneshot command return successful exist code
2023-05-29 10:01:27 +02:00
Micah R Ledbetter
1e50ca243e
Add FAVICON_PROVIDER option for custom favicon service
2023-05-05 20:42:36 -05:00
David Calano
f48e48e6da
Fix for Issue #1008
...
- Added missing decode() when setting pkg_path variable
2023-03-29 01:48:12 -04:00
Tom Ryder
53af810ff8
Add missing closing quote to style attribute
2023-03-27 10:54:04 +13:00
ふぁ
44a5a5ed7e
add explicitly specify --headless=new
...
Signed-off-by: ふぁ <yuki@yuki0311.com>
2023-03-17 19:30:14 +09:00
Nick Sweeting
9f42a3bf29
fix whitespace
2023-03-15 16:01:02 -07:00
ふぁ
d77c770c47
add CHROME_TIMEOUT args
...
Signed-off-by: ふぁ <yuki@yuki0311.com>
2023-03-14 20:29:41 +09:00
Nick Sweeting
606fa397a4
disable passing timeout arg to chrome because v111 is crashing when passed
2023-03-13 10:50:18 +00:00
Nick Sweeting
1f1c70a8b1
remove --single-process from chrome args and add some rendering optimization args
2023-03-13 10:49:57 +00:00
Nick Sweeting
9599845b56
ensure DOM HTML dump is non-zero length file when retrying
2023-03-13 10:49:26 +00:00
Nick Sweeting
dca69933eb
Update archivebox/config.py
...
Co-authored-by: dugite-code <dugite-code@users.noreply.github.com>
2023-01-09 18:22:01 -08:00
Nick Sweeting
2538b170c7
Merge branch 'dev' into feat/reverse-proxy-auth
2023-01-09 18:20:45 -08:00
Nick Sweeting
0cbeeb4346
Merge pull request #1021 from renaisun/dev
2023-01-09 18:17:39 -08:00
Joseph Turian
07de4a79a1
Merge branch 'dev' into feature/kludge-984-UTF8-bug
2022-12-20 11:39:01 +01:00
Nick Sweeting
e114b1f6dc
Merge pull request #1027 from turian/feature/migrations-0021_auto_20220914_0934.py
2022-11-27 19:28:55 -08:00
SnZ
2db830c6a8
Method typo?
...
Fixes '[Errno 2] No such file or directory' error during add
2022-11-20 01:51:16 +01:00
Joseph Turian
a26a91d09f
Merge branch 'feature/migrations-0021_auto_20220914_0934.py' into feature/kludge-984-UTF8-bug
2022-09-14 09:44:55 +00:00
Joseph Turian
22d8e57637
Add missing migration 0021
2022-09-14 09:36:17 +00:00
Joseph Turian
30947aeb07
yt-dlp flag cleanup
2022-09-14 06:29:57 +02:00
Joseph Turian
f729bbe122
yt-dlp fixes
2022-09-14 06:27:58 +02:00
Joseph Turian
081a12b079
Add ts
2022-09-12 21:32:47 +00:00
Joseph Turian
daef48e59b
flake8
2022-09-12 21:31:33 +00:00
