Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2025-02-16 21:38:33 +00:00
Code Issues Projects Releases Packages Wiki Activity
3339 commits 6 branches 79 tags 20 MiB
Commit graph

3339 commits

This branch
This branch
All branches
Author SHA1 Message Date
Ross Williams
310b4d1242 Add htmltotext extractor 
Saves HTML text nodes and selected element attributes in
`htmltotext.txt` for each Snapshot. Primarily intended to be used
for search indexing.
 2023-10-23 21:42:32 -04:00
Nick Sweeting
2681f752f1
ignore playwright chrome install on arm v7 2023-10-21 14:17:50 -07:00
Nick Sweeting
2928c19119 fix multiarch chromium install 2023-10-20 05:45:08 -07:00
Nick Sweeting
988b6585c9 better multiarch builds 2023-10-20 05:30:45 -07:00
Nick Sweeting
25d2c17bb1 add dev build helper script 2023-10-20 05:10:18 -07:00
Nick Sweeting
015737b1e4 fix buildx helper script 2023-10-20 05:10:03 -07:00
Nick Sweeting
bf7a6e4c09 add missing cpython packages needed for arm v8 docker build 2023-10-20 04:33:26 -07:00
Nick Sweeting
63ad43f46c
Merge branch 'dev' into method_allow_deny 2023-10-20 04:25:44 -07:00
Nick Sweeting
53355be46a
Merge branch 'dev' into fts5-search 2023-10-20 04:23:50 -07:00
Nick Sweeting
a58535baff
Merge pull request #1249 from ArchiveBox/pdm 2023-10-20 04:23:25 -07:00
Nick Sweeting
82d8662c74 add more readability error output 2023-10-20 04:14:28 -07:00
Nick Sweeting
f3e81c1487 bump npm lockfile 2023-10-20 04:09:03 -07:00
Nick Sweeting
76f9b91ed3 dockerfile fixes 2023-10-20 04:08:38 -07:00
Nick Sweeting
6736e63ef0 new Dockerfile layout with better layering 2023-10-20 02:47:34 -07:00
Nick Sweeting
86c662421b show FS_USER in version output debug string 2023-10-20 02:47:23 -07:00
Nick Sweeting
7910a5b34b fix pretty_path printing logic 2023-10-20 02:47:03 -07:00
Nick Sweeting
d4ca2d1154 update mercury bin path to postlight parser 2023-10-20 02:46:41 -07:00
Nick Sweeting
75eeb12ebe dont install youtubedl anymore 2023-10-20 02:46:09 -07:00
Nick Sweeting
16796a63fd share PUID and PGID with child procs from entrypoint 2023-10-20 02:45:54 -07:00
Nick Sweeting
e0e34e6377 update package.json and npm lockfile 2023-10-20 02:45:37 -07:00
Nick Sweeting
3b3bdab97d update ignore files 2023-10-20 02:45:20 -07:00
Nick Sweeting
63c276a93d redirect add page back to snapshots list automatically 2023-10-19 18:24:19 -07:00
Nick Sweeting
22bcffe5ee remove accidental duplicate template code 2023-10-19 18:24:05 -07:00
Nick Sweeting
85f8583d62 catch ldap configuration and packaging errors and make them non fatal 2023-10-19 18:05:59 -07:00
Nick Sweeting
53cff45ec0 switch from pipenv and setup.py to PDM for python packaging 2023-10-19 17:49:06 -07:00
Nick Sweeting
8caffc4b4e ignore .pdm-python file 2023-10-19 17:09:48 -07:00
Nick Sweeting
b6113cc00c use pypa build instead of setup.py build and move ldap to extras 2023-10-19 16:44:48 -07:00
Nick Sweeting
4b26ab9d1f only load image into docker when building, and push on deploy 2023-10-19 16:29:51 -07:00
Nick Sweeting
9fa210515e
Bump required python version to 3.9 in stdeb.cfg 2023-10-19 16:22:06 -07:00
Nick Sweeting
748363e685
Bump required Python version to 3.9 2023-10-19 16:21:32 -07:00
Nick Sweeting
6619e8005e
Update pip.yml github action python version 2023-10-19 16:20:43 -07:00
Nick Sweeting
a61f544547
Update SECURITY.md 2023-10-19 16:18:32 -07:00
Nick Sweeting
edde40898b
fix markdown underlining everything as link 2023-10-19 12:58:38 -07:00
Nick Sweeting
a8ce927937
fix markdown formatting 2023-10-19 12:57:05 -07:00
Nick Sweeting
0e3475a0eb
Create SECURITY.md policy 2023-10-18 18:40:58 -07:00
Nick Sweeting
4f655fc4a1
Replace chown of entire data directory when ownership mismatch detected 2023-10-18 18:24:43 -07:00
Nick Sweeting
e9c85aa812
Update README.md 2023-10-18 17:53:49 -07:00
Nick Sweeting
6f4a7e4e68
Add ADMIN_USERNAME and ADMIN_PASSWORD to example options 2023-10-18 11:53:13 -07:00
Nick Sweeting
2a1104c76a
Merge pull request #1245 from benmuth/add-user-env-vars 2023-10-18 11:50:58 -07:00
Nick Sweeting
d286dca925
better to log before doing a thing than after 2023-10-18 11:47:55 -07:00
Ben Muthalaly
9e6a87114b Fix formatting, logging, logic issues 2023-10-18 12:07:36 -05:00
Ben Muthalaly
521ea70e0c Add check for existing user, change varable names 2023-10-18 03:07:54 -05:00
Ross Williams
9b85f35b63 sqlite search: check SQLite version when indexing 
If creating the FTS5 tables fails due to a known version
incompatiblity, report the required version to the user.
 2023-10-16 14:50:48 -04:00
Ross Williams
1e604a1352 sqlite search: clean up errors and type-checking 
Clean up error handling, and report a better error message
on search and flush if FTS5 tables haven't yet been created.

Add some mypy comments to clean up type-checking errors.
 2023-10-16 14:31:52 -04:00
Ross Williams
adb9f0ecc9 sqlite search: Rename connection to get_connection 
`connection` could cause confusion with `django.db.connection` and
`get_connection` is a better callable name.
 2023-10-16 13:23:26 -04:00
Ross Williams
e0f8eeeaa7 Improve search.backends.sqlite retry logic 
Retry with table creation should fail if it is attempted for a second
time.
 2023-10-16 13:23:26 -04:00
Ross Williams
8fe5faf4d0 Introduce SQLite FTS5-powered search backend 
Use SQLite's FTS5 extension to power full-text search without any
additional dependencies. FTS5 was introduced in SQLite 3.9.0,
[released][1] in 2015 so should be available on most SQLite
installations at this point in time.

[1]: https://www.sqlite.org/changes.html#version_3_9_0
 2023-10-16 13:23:26 -04:00
Ross Williams
c53ec45a29 WIP: add sqlite search backend boilerplate 2023-10-16 13:23:26 -04:00
Ross Williams
6555719489 Add space after tags when extracting text 
Add space after any close tag to ensure that
tokens that would be rendered separate in HTML
get extracted as separate tokens in text.

Example:

`<p>First</p><p>Second</p>` --> `First Second`
NOT `FirstSecond`
 2023-10-16 09:59:08 -04:00
Ben Muthalaly
44a94157be Remove logging of configured username and password 2023-10-15 23:36:47 -05:00