Nick Sweeting
b3c1cb716e
move abx plugins inside vendor dir
2024-10-28 04:07:35 -07:00
Nick Sweeting
80d8a6b667
split archivebox.use into archivebox.reads and archivebox.writes
2024-10-15 01:03:01 -07:00
Nick Sweeting
f75ae805f8
comment out Crawl api methods temporarily
2024-10-14 15:41:58 -07:00
Nick Sweeting
cf1ea8f80f
improve config loading of TMP_DIR, LIB_DIR, move to separate files
2024-10-07 23:45:11 -07:00
Nick Sweeting
5697ecefad
fix index SEARCH_BACKENDS import_backend to load via ABX instead of settings
2024-10-01 00:19:19 -07:00
Nick Sweeting
d21bc86075
finish migrating almost all config to new system
2024-09-30 23:21:34 -07:00
Nick Sweeting
363a499289
move util.py into misc folder
2024-09-30 17:25:15 -07:00
Nick Sweeting
3e5b6ddeae
move config into dedicated global app
2024-09-30 15:59:05 -07:00
Nick Sweeting
64c7100cf9
speed up startup time, add rich startup progressbar, split logging and checks into misc, fix search index import backend bug
2024-09-24 19:04:38 -07:00
Nick Sweeting
fbfd16e195
fully migrate all search backends to new plugin system
2024-09-24 03:05:43 -07:00
Nick Sweeting
c9c163efed
begin migrating search backends to new plugin system
2024-09-24 02:13:01 -07:00
Nick Sweeting
0420662174
switch everywhere to use Snapshot.pk and ArchiveResult.pk instead of id
2024-05-13 05:12:12 -07:00
Ross Williams
9b85f35b63
sqlite search: check SQLite version when indexing
...
If creating the FTS5 tables fails due to a known version
incompatiblity, report the required version to the user.
2023-10-16 14:50:48 -04:00
Ross Williams
1e604a1352
sqlite search: clean up errors and type-checking
...
Clean up error handling, and report a better error message
on search and flush if FTS5 tables haven't yet been created.
Add some mypy comments to clean up type-checking errors.
2023-10-16 14:31:52 -04:00
Ross Williams
adb9f0ecc9
sqlite search: Rename connection to get_connection
...
`connection` could cause confusion with `django.db.connection` and
`get_connection` is a better callable name.
2023-10-16 13:23:26 -04:00
Ross Williams
e0f8eeeaa7
Improve search.backends.sqlite retry logic
...
Retry with table creation should fail if it is attempted for a second
time.
2023-10-16 13:23:26 -04:00
Ross Williams
8fe5faf4d0
Introduce SQLite FTS5-powered search backend
...
Use SQLite's FTS5 extension to power full-text search without any
additional dependencies. FTS5 was introduced in SQLite 3.9.0,
[released][1] in 2015 so should be available on most SQLite
installations at this point in time.
[1]: https://www.sqlite.org/changes.html#version_3_9_0
2023-10-16 13:23:26 -04:00
Ross Williams
c53ec45a29
WIP: add sqlite search backend boilerplate
2023-10-16 13:23:26 -04:00
Nick Sweeting
3093057675
bail out on sonic indexing after 5 errors
2021-04-10 05:18:03 -04:00
Nick Sweeting
32764347ce
add new SEARCH_BACKEND_TIMEOUT config option defaulted to 90sec
2021-04-09 00:27:12 -04:00
Nick Sweeting
21bb84f627
fix lint errors
2021-04-06 02:50:51 -04:00
Nick Sweeting
f67a5a215a
fix readability indexing process and implement a max total character length on indexed content
2021-04-06 02:01:38 -04:00
Nick Sweeting
bd6d9c165b
enforce utf8 on literally all file operations because windows sucks
2021-03-27 01:16:29 -04:00
Nick Sweeting
24e24934f7
add headers.json and fix relative singlefile path resolving for sonic
2021-01-30 21:59:34 -05:00
jdcaballerov
14df0cbb7c
Update sonic.py
...
Sonic buffer accepts 20.000 bytes not unicode characters, since the chunking here is on unicode characters, sending 20.000 characters will overflow sonic's buffer.
UTF-8 can take up to 6 bytes, so sending less than (20.000 / 6) rounded minus should be ok.
2021-01-20 14:51:46 -05:00
Nick Sweeting
326fe69eea
fix lint error
2020-12-12 12:35:32 -05:00
jdcaballerov
9b6afa36a3
Update archivebox/search/backends/ripgrep.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-12-12 08:36:08 -05:00
jdcaballerov
aa53f4f088
Update archivebox/search/backends/ripgrep.py
...
Co-authored-by: Nick Sweeting <git@sweeting.me>
2020-12-12 08:36:01 -05:00
jdcaballerov
24d4c44624
Add ripgrep configs
2020-12-12 07:36:31 -05:00
Cristian
e82161a768
refactor: Remove setup_django from search
2020-12-11 16:43:48 -05:00
Nick Sweeting
e90cf05141
fix lint errors
2020-12-11 16:51:11 +02:00
Cristian
9aac09a5e1
feat: Patch setup_django so we can use an inmemory db in specific commands
2020-12-08 18:42:25 -05:00
Cristian
8d22ebf988
feat: Remove walrus operator (we still need to support python3.7)
2020-12-06 12:23:02 -05:00
jdcaballerov
172197ae01
refactor: Remove if LENGTH and use text chunker for every input
2020-12-06 01:14:39 +02:00
jdcaballerov
5a6b814c79
Add exception handling for indexable content reader
2020-12-06 01:14:38 +02:00
JDC
15fbd81480
Change MAX_SONIC_TEXT_LENGTH
2020-12-06 01:14:38 +02:00
JDC
db9c2edccc
Add log print for url indexing
2020-12-06 01:14:38 +02:00
JDC
0acf479b70
Partition long strings in chunks for sonic
2020-12-06 01:14:38 +02:00
JDC
caf4660ac8
Add indexing to update command and utilities
2020-12-06 01:14:37 +02:00
JDC
23a9beb4e0
Add ignored extensions in ripgrep search
2020-12-06 01:13:39 +02:00
JDC
95382b3812
Add ripgrep rg search backend and set as default
2020-12-06 01:13:39 +02:00
JDC
4eeedae815
Exception handling for indexing and searching
2020-12-06 01:13:39 +02:00
JDC
fb67d6684c
fix: Return empty QuerySet instead of list
2020-12-06 01:12:47 +02:00
JDC
823df34080
Use QuerySets for search backend API instead of pks
2020-12-06 01:12:47 +02:00
JDC
f383648ffc
Use a generator for snapshot flush from index
2020-12-06 01:12:47 +02:00
JDC
47daa038eb
Implement flush for search backend after remove command
2020-12-06 01:12:47 +02:00
JDC
c2c01af3ad
Add config for search backend
2020-12-06 01:12:47 +02:00
JDC
5f6673c72c
Implement backend architecture for search engines
2020-12-06 01:12:46 +02:00
JDC
b1f70b2197
Initial implementation
2020-12-06 01:12:45 +02:00
