Commit graph

1996 commits

Author SHA1 Message Date
Nick Sweeting
b6e60e63bc
bump pydantic pkgr 2024-10-24 15:12:44 -07:00
Nick Sweeting
9e40dd69a4
more config improvements, move away from settings GLOBALS to getters 2024-10-24 14:50:07 -07:00
Nick Sweeting
b9492f7736
silence ABID drift warning 2024-10-24 14:50:07 -07:00
Nick Sweeting
082930b2ef
add wip states 2024-10-22 02:25:54 -07:00
Nick Sweeting
72f777059f
minor plugin improvements 2024-10-22 02:25:20 -07:00
Nick Sweeting
1dff8bae40
dont try to install brew casks on linux 2024-10-21 03:44:45 -07:00
Nick Sweeting
cd3b937469
fix syntax error 2024-10-21 03:21:48 -07:00
Nick Sweeting
657eec479b
fix CONSTANTS.LIB_DIR old style access 2024-10-21 03:20:20 -07:00
Nick Sweeting
9b7ad2cae6
fix Dockerfile and INSTALLER_BIN_ABSPATH when not available on host 2024-10-21 03:16:07 -07:00
Nick Sweeting
8a2b38b46c
tweak version output order 2024-10-21 03:12:48 -07:00
Nick Sweeting
312e40b95b
finally get rid of config/legacy in favor of configfile.py and django.py 2024-10-21 03:06:19 -07:00
Nick Sweeting
b3107ab830
move final legacy config to plugins and fix archivebox config cmd and add search opt 2024-10-21 02:56:00 -07:00
Nick Sweeting
5e743b6aee
create Chrome profile on first run automagically 2024-10-21 01:33:08 -07:00
Nick Sweeting
267fde0138
call validate functions at AppConfig.ready time manually 2024-10-21 01:32:53 -07:00
Nick Sweeting
7a6f1f36d2
trigger abx.pm.hook.ready from core.AppConfig.ready 2024-10-21 01:31:02 -07:00
Nick Sweeting
dd40f252ed
dont warn about update_in_place config more than once per key 2024-10-21 01:30:42 -07:00
Nick Sweeting
5023bdba2f
only list admins that are not system user 2024-10-21 00:38:36 -07:00
Nick Sweeting
a97cc82979
fix Crawl models 2024-10-21 00:38:29 -07:00
Nick Sweeting
d48a99ab64
only load django huey config when not getting help or version to avoid creating queue.sqlite3 2024-10-21 00:38:17 -07:00
Nick Sweeting
02a1fc3049
rename sessions app in INSTALLED_APPS to personas 2024-10-21 00:37:57 -07:00
Nick Sweeting
82db227561
move sessions app to personas to avoid app_label conflict 2024-10-21 00:36:17 -07:00
Nick Sweeting
a211461ffc
fix LIB_DIR and TMP_DIR loading when primary option isnt available 2024-10-21 00:35:56 -07:00
Nick Sweeting
35dd5911d8
WIP add new effects, reads, writes, to abx interface 2024-10-15 22:33:22 -07:00
Nick Sweeting
0abbc11a6b
add get_scope_config method to merge config down to scope 2024-10-15 22:32:36 -07:00
Nick Sweeting
c9f88f5875
add sessions and seeds models 2024-10-15 22:32:13 -07:00
Nick Sweeting
0866f4aaf3
add architecture mockup 2024-10-15 01:03:17 -07:00
Nick Sweeting
80d8a6b667
split archivebox.use into archivebox.reads and archivebox.writes 2024-10-15 01:03:01 -07:00
Nick Sweeting
aaf069fab0
remove tags field from Machine admin 2024-10-15 01:02:13 -07:00
Nick Sweeting
df79b8e038
rename config sections to match old sections 2024-10-15 01:01:34 -07:00
Nick Sweeting
7671813aa5
enable relevant hookspecs 2024-10-15 01:01:23 -07:00
Nick Sweeting
c04e0b2f39
add abx function to get plugin hooks 2024-10-15 01:01:10 -07:00
Nick Sweeting
8d87378a46
better docstrings in abx 2024-10-15 01:00:57 -07:00
Nick Sweeting
536e3ac3c4
make textarea django-jsonfield look nicer 2024-10-15 01:00:42 -07:00
Nick Sweeting
654bd22bd0
re-author plugins 2024-10-15 01:00:21 -07:00
Nick Sweeting
01ba6d49d3
new vastly simplified plugin spec without pydantic 2024-10-14 21:50:47 -07:00
Nick Sweeting
abf75f49f4
add tentative handling for Hooks as modules instead of BaseHook subclasses 2024-10-14 17:42:52 -07:00
Nick Sweeting
d69df359ea
remove Crawl migration in favor of separate app 2024-10-14 17:41:07 -07:00
Nick Sweeting
943f843189
add core/admin_site 2024-10-14 17:40:35 -07:00
Nick Sweeting
bb9c3fda14
fix makemigrations being blocked by check_migrations func 2024-10-14 17:40:06 -07:00
Nick Sweeting
2ebd28aebd
fix ABID and uniqueness for new Seed models 2024-10-14 17:39:43 -07:00
Nick Sweeting
59b669691f
fix Admin data view for Config to render both sections and individual values 2024-10-14 17:39:14 -07:00
Nick Sweeting
1d7f0ab20d
fix Tag creation via admin erroring because slug field is not filled 2024-10-14 17:38:53 -07:00
Nick Sweeting
c0b7887fd7
fix admin registration using abx hooks 2024-10-14 17:38:38 -07:00
Nick Sweeting
30923c340f
move mixins.py for search into search app 2024-10-14 17:37:30 -07:00
Nick Sweeting
8cff6ddfc6
move ldap_auth setup into LDAP plugin 2024-10-14 17:36:45 -07:00
Nick Sweeting
86380a1ef2
fix .archivebox_id being created outside collection dir 2024-10-14 17:35:43 -07:00
Nick Sweeting
9a04ed7c76
move serve_static and shell_welcome_message into misc 2024-10-14 17:35:28 -07:00
Nick Sweeting
5ac941cf2e
resolve circular import in core/views 2024-10-14 15:43:53 -07:00
Nick Sweeting
a0bef4e27b
move crawl model out of core 2024-10-14 15:42:36 -07:00
Nick Sweeting
f75ae805f8
comment out Crawl api methods temporarily 2024-10-14 15:41:58 -07:00
Nick Sweeting
536a5ea745
clear up Machine models cache vars 2024-10-14 15:38:32 -07:00
Nick Sweeting
b3b2c551c2
fix get_sock_file symlink process 2024-10-14 15:37:48 -07:00
Nick Sweeting
a7c19a5da0
add register_admin to abx hookspec 2024-10-14 15:35:44 -07:00
Nick Sweeting
4634cd6c8e
make should_extract take config and uri as args 2024-10-14 15:35:20 -07:00
Nick Sweeting
518c46b4ab
fix circular import and show log of plugins loading on startup 2024-10-14 15:35:03 -07:00
Nick Sweeting
ad4657861f
add new crawls app 2024-10-11 19:47:05 -07:00
Nick Sweeting
1768f147d4
fix playwright and puppeteer install methods on linux 2024-10-11 02:21:15 -07:00
Nick Sweeting
38cb1ac87b
add apt install method for chrome 2024-10-11 02:20:48 -07:00
Nick Sweeting
f12efbdb45
use new dry run system 2024-10-11 02:20:37 -07:00
Nick Sweeting
bffa9ee78d
fix puppeteer install using npx stalling waiting for y user confirmation 2024-10-11 01:03:34 -07:00
Nick Sweeting
146cdff7d9
fix apt not installing npm package as separate package 2024-10-11 01:03:21 -07:00
Nick Sweeting
6e7071bd19
add new binproviders and binaries args to install and version, bump pydantic-pkgr version 2024-10-11 00:45:59 -07:00
Nick Sweeting
5e2dfb5baa
totally ignore sonic unless its the selected search backend 2024-10-09 19:12:47 -07:00
Nick Sweeting
8ce7313325
ignore puppeteer and playwright install stdout lines that arent chrome bin abspath 2024-10-09 19:12:35 -07:00
Nick Sweeting
0c29e08f73
avoid creating collection id file on every startup since its not needed 2024-10-09 19:12:08 -07:00
Nick Sweeting
b1a17689cd
dont load sonic binary if sonic backend is not enabled 2024-10-09 18:57:24 -07:00
Nick Sweeting
c25ced0c2a
fix Node binary to install with apt as nodejs 2024-10-09 18:40:50 -07:00
Nick Sweeting
228bb7bd52
disable LDAP_BINARY if LDAP_ENABLED=False 2024-10-09 18:39:45 -07:00
Nick Sweeting
fdc720b13d
remove checks import and bump pydantic pkgr version 2024-10-09 18:28:05 -07:00
Nick Sweeting
04d2316800
remove CHECKS feature, not needed 2024-10-09 18:06:00 -07:00
Nick Sweeting
de7ab65f11
ignore errors when chowning at initial startup 2024-10-09 04:48:09 -07:00
Nick Sweeting
ad675a8e7c
properly handle chowning DATA_DIR on init when using sudo 2024-10-09 04:39:09 -07:00
Nick Sweeting
2f68a1d476
fix ldap lib loading after apt install 2024-10-09 04:03:02 -07:00
Nick Sweeting
1b7aca130b
properly detect sudo UID 2024-10-09 04:02:46 -07:00
Nick Sweeting
31c66a1068
fix playwright and puppeteer browser detection after install 2024-10-09 03:46:28 -07:00
Nick Sweeting
f4f9ae2f28
fix LIB_NPM provider loading and euid 2024-10-09 03:19:31 -07:00
Nick Sweeting
861b9cd16f
fix LDAP_LIB loading from venv and other site packages dirs 2024-10-09 03:19:09 -07:00
Nick Sweeting
afc24e802a
tweak version log output 2024-10-09 03:18:22 -07:00
Nick Sweeting
db65af898b
correctly update environment HOME and USER vars when dropping permissions 2024-10-09 03:18:04 -07:00
Nick Sweeting
48409d8a5a
run env BinProvider as ArchiveBox User 2024-10-09 03:17:50 -07:00
Nick Sweeting
613caec8eb
improve install flow with sudo, check package managers, and fix docker build 2024-10-09 00:41:16 -07:00
Nick Sweeting
e4f543f54a
add new plugin order sigil file feature 2024-10-09 00:39:46 -07:00
Nick Sweeting
f6446a141f
also try installing ldap with apt if its not available 2024-10-08 19:27:02 -07:00
Nick Sweeting
7c34f2bc90
hide errors if user is just getting help or version info 2024-10-08 19:20:03 -07:00
Nick Sweeting
9f274cf9f4
remove platformdirs dependency 2024-10-08 19:17:18 -07:00
Nick Sweeting
0b4cbb6415
clear out stale supervisord state in between runs 2024-10-08 18:59:48 -07:00
Nick Sweeting
584abe8548
never attempt to create system venv, install ldap in lib automatically, and setup binproviders before bins 2024-10-08 18:52:02 -07:00
Nick Sweeting
3e4a846488
fix more installer bugs 2024-10-08 18:06:57 -07:00
Nick Sweeting
4b34b729ab
fuck it go back to nested lib and tmp dirs with supervisord sock workaround 2024-10-08 17:48:59 -07:00
Nick Sweeting
1888691ee8
try creating shared libs as 777 when running as root 2024-10-08 17:10:56 -07:00
Nick Sweeting
35c7019772
handle failure on tmp_dir and lib_dir detection better 2024-10-08 16:56:25 -07:00
Nick Sweeting
a33da44492
more attempts to fix euid permissions issues on ubuntu 2024-10-08 16:56:24 -07:00
Nick Sweeting
216e885b85
bump pydantic-pkgr 2024-10-08 03:53:41 -07:00
Nick Sweeting
8d32508581
only show data dir info if one is active 2024-10-08 03:04:38 -07:00
Nick Sweeting
de2ab43f7f
switch .is_dir and .exists for os.access to avoid PermissionError on startup 2024-10-08 03:02:34 -07:00
Nick Sweeting
611a2b7c1b
fix a few small nits 2024-10-08 02:10:08 -07:00
Nick Sweeting
3fb5b6eb94
exit archivebox version with failure status if any subdependencies are not installed 2024-10-08 01:52:04 -07:00
Nick Sweeting
5e351f6ba6
more docker dependency tweaks 2024-10-08 01:47:38 -07:00
Nick Sweeting
46c0463539
safer import handling 2024-10-08 00:51:58 -07:00
Nick Sweeting
d9fef4cd84
fix import order 2024-10-08 00:15:53 -07:00
Nick Sweeting
397ae1a99b
fix Docker build and import issues 2024-10-08 00:12:09 -07:00
Nick Sweeting
cf1ea8f80f
improve config loading of TMP_DIR, LIB_DIR, move to separate files 2024-10-07 23:45:11 -07:00
Nick Sweeting
7a895d9285
make sure monkey patches and vendor libs come before first constants import 2024-10-05 16:42:51 -07:00
Nick Sweeting
430e974719
add pydantic-pkgr back as vendored lib submodule for now 2024-10-05 16:39:20 -07:00
Nick Sweeting
ccdc3e1c47
add pydantic-pkgr back as vendored lib becauase pypi is misbehaving 2024-10-05 16:38:55 -07:00
Nick Sweeting
55e286972d
fix timeout check showing regardless of value 2024-10-05 04:24:07 -07:00
Nick Sweeting
db10a2142e
remove extra files from repo root and move package.json into etc 2024-10-05 03:53:23 -07:00
Nick Sweeting
66a785bb35
only use system tmp dirs because of socket path length restrictions 2024-10-05 03:16:27 -07:00
Nick Sweeting
35446ce742
include sonic-client by default and allow ldap to be installed at runtime 2024-10-05 03:11:48 -07:00
Nick Sweeting
ce2e19a429
switch to uv builds and rc1 versioning system 2024-10-04 23:48:25 -07:00
Nick Sweeting
0876cc78d9
remove no longer needed vendored libs 2024-10-04 23:35:34 -07:00
Nick Sweeting
80e052b166
fix pip binary loading 2024-10-04 23:26:49 -07:00
Nick Sweeting
beefe69b74
fix CHROME_TIMEOUT causing hanging on some platforms 2024-10-04 21:49:09 -07:00
Nick Sweeting
ac96cc62fc
fix CUSTOM_TEMPLATES_DIR loading 2024-10-04 21:40:36 -07:00
Nick Sweeting
0c7d7a2225
fix archivebox init colors and dir status checking 2024-10-04 21:34:19 -07:00
Nick Sweeting
5323953f94
handle Ctrl+C more gracefully 2024-10-04 21:33:46 -07:00
Nick Sweeting
026169a8e2
fix rich colors 2024-10-04 21:09:29 -07:00
Nick Sweeting
d747cf7f31
fix SYSTEM_TMP_DIR and SYSTEM_LIB_DIR in docker 2024-10-04 21:03:02 -07:00
Nick Sweeting
811f9a8d93
move queue db name into constants and fix file detection at startup 2024-10-04 19:38:36 -07:00
Nick Sweeting
3f986f09cc
fix relative dir calculation in extraactor hook 2024-10-04 19:24:01 -07:00
Nick Sweeting
73e69ccb8b
fixes for docs generation 2024-10-04 19:16:46 -07:00
Nick Sweeting
f76bdc4332
fix old wsgi.py 2024-10-04 16:08:57 -07:00
Nick Sweeting
da274fd8e8
remove dead code 2024-10-04 14:48:20 -07:00
Nick Sweeting
396a7ffcd8
move tmp dir to machine-id scoped dir 2024-10-04 03:24:15 -07:00
Nick Sweeting
12f32c4690
fix tmp data dir resolution when running help or version outside data dir 2024-10-04 01:40:41 -07:00
Nick Sweeting
f321d25f4c
fallback to reading binaries from filesystem when theres no db 2024-10-04 01:00:09 -07:00
Nick Sweeting
152b530249
scope LIB_DIR by os, arch, and docker status 2024-10-04 00:08:44 -07:00
Nick Sweeting
c84ea81c5a
more ldap lib optimization 2024-10-03 18:25:35 -07:00
Nick Sweeting
89a066da0b
remove django-url-tools in favor of core_tags snippet 2024-10-03 18:25:20 -07:00
Nick Sweeting
3f1a19dd35
fix type imports 2024-10-03 18:24:57 -07:00
Nick Sweeting
c5da3c1f22
fix docker build 2024-10-03 18:24:27 -07:00
Nick Sweeting
0619750ffa
add django-url-tools to fix pagination and search on public index 2024-10-03 17:39:55 -07:00
Nick Sweeting
1492c02bfa
lazy-load loadfire and ldap lib for faster startup time 2024-10-03 17:39:39 -07:00
Nick Sweeting
563e4de678
unwinding circular dependencies 2024-10-03 17:05:49 -07:00
Nick Sweeting
9241a45bb8
Update base_configset.py 2024-10-03 11:42:27 -04:00
Nick Sweeting
de09867f87
ignore lib/bin symlinking errors 2024-10-03 04:10:52 -07:00
Nick Sweeting
aae9deccc2
add machine migrations 2024-10-03 04:06:28 -07:00
Nick Sweeting
b072fd8ef4
load all binaries from cache by default 2024-10-03 04:06:17 -07:00
Nick Sweeting
0f37abb657
fix symlinking to lib when conflicting file already exists 2024-10-03 03:56:45 -07:00
Nick Sweeting
490e5ba11d
fallback to localhost if detecting dnsserver fails 2024-10-03 03:53:50 -07:00
Nick Sweeting
b36e89d086
relocate LIB_DIR and TMP_DIR inside docker so it doesnt clash with outside docker 2024-10-03 03:43:02 -07:00
Nick Sweeting
f4f1d7893c
fix CUSTOM_TEMPLATES_DIR config load and chrome symlinking 2024-10-03 03:20:25 -07:00
Nick Sweeting
29fc14dff4
dont build docker container twice during release 2024-10-03 03:12:18 -07:00
Nick Sweeting
9728d81fee
add puppeteer to docker requirements for easier browser fetching 2024-10-03 03:12:06 -07:00
Nick Sweeting
697d0a3566
nicer version and help pretty printing with rich 2024-10-03 03:11:23 -07:00
Nick Sweeting
161afc7297
add health stats counters to machine models 2024-10-03 03:11:04 -07:00
Nick Sweeting
3b9e48ead8
show deprecation warning for archivebox setup command 2024-10-03 03:10:36 -07:00
Nick Sweeting
e315905721
add new InstalledBinary model to cache binaries on host machine 2024-10-03 03:10:22 -07:00
Nick Sweeting
32167de936
add daemonize flag to archivebox server 2024-10-02 19:46:48 -07:00
Nick Sweeting
035a14b6ea
better help text output 2024-10-02 19:46:31 -07:00