Nick Sweeting
f6ea21eabf
bump version in package.json
2023-11-14 00:52:29 -08:00
Nick Sweeting
6188dcdcbd
return placeholder version during docs build
2023-11-14 00:38:37 -08:00
Nick Sweeting
d3d05f5f64
add fallback version detection mode
2023-11-14 00:21:09 -08:00
Nick Sweeting
4523d6817f
readme fixes
2023-11-13 23:51:46 -08:00
Nick Sweeting
716f68c34a
bump vendor dependency versions
2023-11-13 22:38:19 -08:00
Nick Sweeting
f5739506f6
Merge pull request #1244 from overhacked/search_index_extract_html_text
2023-11-08 23:34:44 -08:00
Nick Sweeting
7377d9ebc3
bump version to 0.7.1 and fix version autodetection code
2023-11-03 20:07:39 -07:00
Nick Sweeting
db733151f7
bump user agent browser version
2023-10-31 19:16:15 -07:00
Nick Sweeting
166573d417
autodetect qemu and warn about broken chrome in emulated containers
2023-10-31 00:23:19 -07:00
Nick Sweeting
62e077a5bc
Merge pull request #1241 from overhacked/fts5-search
2023-10-30 18:49:52 -07:00
Nick Sweeting
851f59b4ba
Merge pull request #1254 from c01o/dev
2023-10-30 18:44:55 -07:00
mAAdhaTTah
18e89bee3d
Fix readwise token
...
Accidentally shipped this hard-coded. Token is long-size revoked,
but obviously hard to use this way!
2023-10-29 17:27:04 -04:00
Nick Sweeting
a680724367
Merge branch 'dev' into search_index_extract_html_text
2023-10-27 23:09:28 -07:00
c01o
8c25a81009
Remove playwright from armv7
2023-10-27 05:59:25 +09:00
Ross Williams
310b4d1242
Add htmltotext extractor
...
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
2023-10-23 21:42:32 -04:00
Nick Sweeting
63ad43f46c
Merge branch 'dev' into method_allow_deny
2023-10-20 04:25:44 -07:00
Nick Sweeting
53355be46a
Merge branch 'dev' into fts5-search
2023-10-20 04:23:50 -07:00
Nick Sweeting
82d8662c74
add more readability error output
2023-10-20 04:14:28 -07:00
Nick Sweeting
76f9b91ed3
dockerfile fixes
2023-10-20 04:08:38 -07:00
Nick Sweeting
86c662421b
show FS_USER in version output debug string
2023-10-20 02:47:23 -07:00
Nick Sweeting
7910a5b34b
fix pretty_path printing logic
2023-10-20 02:47:03 -07:00
Nick Sweeting
d4ca2d1154
update mercury bin path to postlight parser
2023-10-20 02:46:41 -07:00
Nick Sweeting
63c276a93d
redirect add page back to snapshots list automatically
2023-10-19 18:24:19 -07:00
Nick Sweeting
22bcffe5ee
remove accidental duplicate template code
2023-10-19 18:24:05 -07:00
Nick Sweeting
85f8583d62
catch ldap configuration and packaging errors and make them non fatal
2023-10-19 18:05:59 -07:00
Nick Sweeting
d286dca925
better to log before doing a thing than after
2023-10-18 11:47:55 -07:00
Ben Muthalaly
9e6a87114b
Fix formatting, logging, logic issues
2023-10-18 12:07:36 -05:00
Ben Muthalaly
521ea70e0c
Add check for existing user, change varable names
2023-10-18 03:07:54 -05:00
Ross Williams
9b85f35b63
sqlite search: check SQLite version when indexing
...
If creating the FTS5 tables fails due to a known version
incompatiblity, report the required version to the user.
2023-10-16 14:50:48 -04:00
Ross Williams
1e604a1352
sqlite search: clean up errors and type-checking
...
Clean up error handling, and report a better error message
on search and flush if FTS5 tables haven't yet been created.
Add some mypy comments to clean up type-checking errors.
2023-10-16 14:31:52 -04:00
Ross Williams
adb9f0ecc9
sqlite search: Rename connection to get_connection
...
`connection` could cause confusion with `django.db.connection` and
`get_connection` is a better callable name.
2023-10-16 13:23:26 -04:00
Ross Williams
e0f8eeeaa7
Improve search.backends.sqlite retry logic
...
Retry with table creation should fail if it is attempted for a second
time.
2023-10-16 13:23:26 -04:00
Ross Williams
8fe5faf4d0
Introduce SQLite FTS5-powered search backend
...
Use SQLite's FTS5 extension to power full-text search without any
additional dependencies. FTS5 was introduced in SQLite 3.9.0,
[released][1] in 2015 so should be available on most SQLite
installations at this point in time.
[1]: https://www.sqlite.org/changes.html#version_3_9_0
2023-10-16 13:23:26 -04:00
Ross Williams
c53ec45a29
WIP: add sqlite search backend boilerplate
2023-10-16 13:23:26 -04:00
Ross Williams
6555719489
Add space after tags when extracting text
...
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.
Example:
`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
2023-10-16 09:59:08 -04:00
Ben Muthalaly
44a94157be
Remove logging of configured username and password
2023-10-15 23:36:47 -05:00
Ben Muthalaly
11d473e536
Add config options to add admin user on first run
2023-10-14 00:38:04 -05:00
Ross Williams
d8aa84ac98
Make extracting text for indexing optional
...
Add a configuration option to enable/disable HTML text extraction
for indexing
2023-10-12 13:14:39 -04:00
Ross Williams
b6a20c962a
Extract text from singlefile.html when indexing
...
singlefile.html contains a lot of large strings in the form of `data:`
URLs, which can be unnecessarily stored in full-text indices. Also,
large chunks of JavaScript shouldn't be indexed, either, as they pollute
search results for searches about JS functions, etc.
This commit takes a blanket approach of parsing singlefile.html as it is
read and only outputting text and selected textual attributes (like
`alt`) for indexing.
2023-10-12 13:06:35 -04:00
Ben Muthalaly
77917e9b55
Fix HTML title parsing bugs.
...
This slightly modifies the HTML_TITLE_REGEX to fix two parsing errors.
The first occurred when title tags were empty (e.g. "<title></title>")
which was parsed as "</title". The second occurred when titles were a
single character (e.g. "<title>A</title>") which was not matched by the
regex, and so would fall back to link.base_url.
Now when tags are empty, it falls back to link.base_url, and single
character titles are parsed correctly.
The way the regex works now is still a bit wonky for some edge cases.
I couldn't find any cases of incorrect behavior, but it still might be
worth reworking more completely for robustness.
2023-10-09 02:00:01 -05:00
Nick Sweeting
5c1a14e4f2
ignore errors while getting system user name
2023-09-14 03:39:44 -07:00
Nick Sweeting
ffe2968e4f
improve some comments
2023-09-14 02:41:27 -07:00
Nick Sweeting
f809efce4d
Merge pull request #996 from barthalion/dev
2023-09-03 21:40:49 -07:00
Nick Sweeting
aaca74f6a8
only start parsing json after the first open brace
2023-09-03 21:40:12 -07:00
Nick Sweeting
cd9f228b2f
Merge pull request #1214 from DanielBatteryStapler/DanielBatteryStapler-patch-1
2023-09-03 21:25:12 -07:00
Nick Sweeting
16d278fbdb
Merge pull request #1168 from mAAdhaTTah/add-readwise-reader
2023-09-03 21:24:49 -07:00
Nick Sweeting
110a22ee32
Merge branch 'dev' into DanielBatteryStapler-patch-1
2023-08-31 15:20:46 -07:00
Nick Sweeting
73a5f74d38
update default YOUTUBEDL_ARGS to fix subs and filesize
2023-08-31 15:17:45 -07:00
Nick Sweeting
86366d5640
Update logging_util.py to fix generator subscripting error
2023-08-31 15:12:43 -07:00
spresse1
603ce7ec10
After a timeout, chrome will leave behind a SingletonLock, which prevents future instances of chrome from starting. When an extractor fails due to a timeout, remove this file.
2023-08-28 17:27:03 +02:00
