Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-10 06:34:16 +00:00
Code Issues Projects Releases Packages Wiki Activity
4467 commits 6 branches 77 tags 18 MiB
Commit graph

1925 commits

Author SHA1 Message Date
Nick Sweeting
7c363bffc6
add ini_to_toml test 2024-09-24 19:04:54 -07:00
Nick Sweeting
64c7100cf9
speed up startup time, add rich startup progressbar, split logging and checks into misc, fix search index import backend bug 2024-09-24 19:04:38 -07:00
Nick Sweeting
7ffb81f61b
delete dead code 2024-09-24 15:26:43 -07:00
Nick Sweeting
97695bda5e
more settings loading tweaks and improvements 2024-09-24 15:13:54 -07:00
Nick Sweeting
fbfd16e195
fully migrate all search backends to new plugin system 2024-09-24 03:05:43 -07:00
Nick Sweeting
c9c163efed
begin migrating search backends to new plugin system 2024-09-24 02:13:01 -07:00
Nick Sweeting
2d19317e3f
rename plugins_sys base to config 2024-09-24 02:12:30 -07:00
Nick Sweeting
e8f1264954
rename plugins dirs 2024-09-24 01:34:27 -07:00
Nick Sweeting
8713091e73
remove redundant import 2024-09-24 01:32:01 -07:00
Nick Sweeting
77d3990535
temporarily add prints on plugin setup for easier debugging 2024-09-24 01:26:16 -07:00
Nick Sweeting
a9a97c013d
split plugin dirs, created new cleaner import path for plugin config in settings.py 2024-09-24 01:25:55 -07:00
Nick Sweeting
1a58967e8c
first example of plugin config based on another plugin config 2024-09-23 21:10:19 -07:00
Nick Sweeting
8df9480824
make sure hooks have the object identity everywhere in the codebase by avoiding pydantics usual deepcopy on every validation 2024-09-23 21:04:23 -07:00
Nick Sweeting
4eb1c14139
handle ConfigSet default value factories that dont take any args 2024-09-23 21:03:16 -07:00
Nick Sweeting
1f4cded152
use benedict in old config instead of AttrDict 2024-09-23 21:02:51 -07:00
Nick Sweeting
e992a84b80
add custom TOML encoder to work around issues with dumping toml of lots of different types 2024-09-23 21:02:33 -07:00
Nick Sweeting
b6cfeb8d40
add new pydantic_settings based loader for ConfigSets 2024-09-22 19:30:24 -07:00
Nick Sweeting
c8ff8f2b86
add header to generated TOML file warning its been converted from INI 2024-09-22 19:27:33 -07:00
Nick Sweeting
7f05026022
change is_registered and is_ready into private model fields 2024-09-22 19:27:00 -07:00
Nick Sweeting
8f38f70e4a
define PACKAGE_DIR and DATA_DIR in settings.py directly 2024-09-22 19:26:26 -07:00
Nick Sweeting
8c8c64d90f
swap AttrDict for benedict everywhere 2024-09-22 19:26:05 -07:00
Nick Sweeting
b611c0114c
add pydantic_settings mockup 2024-09-22 16:48:28 -07:00
Nick Sweeting
3b0a25950d
add minor pydantic pkgr fix 2024-09-22 16:28:48 -07:00
Nick Sweeting
d89b6ce419
add SQLite semaphore mockup 2024-09-22 16:28:30 -07:00
Nick Sweeting
a2d827afd6
bump pydantic pkgr to 0.3.5 2024-09-22 15:41:21 -07:00
Nick Sweeting
f8c6ff88ad
add clickable host link back to archivebox server output 2024-09-22 15:41:21 -07:00
Nick Sweeting
2d99f184d3
add mockup for new config loading process 2024-09-22 15:41:21 -07:00
Nick Sweeting
ab0087e106
cleanup chrome and playwright symlink and app names 2024-09-22 15:41:20 -07:00
Nick Sweeting
8945475f8d
bump pydantic-pkgr submodule to 0.3.4 2024-09-21 04:12:59 -07:00
Nick Sweeting
99dd812e3b
bump pydantic-pkgr version to 0.3.4 2024-09-21 04:12:34 -07:00
Nick Sweeting
541cd6c5a1
split puppeteer plugin into Puppeteer, Playwright, and Chrome 2024-09-21 04:12:34 -07:00
Nick Sweeting
33fd7fe439
fix log_list_view trying to seek past end of file on short logs 2024-09-21 04:12:34 -07:00
Nick Sweeting
aa21c56ddd
add timeout limit to bin_version loading in config 2024-09-21 04:12:34 -07:00
Nick Sweeting
575105006d
add LIB_DIR and BIN_DIR to config 2024-09-21 04:12:34 -07:00
Nick Sweeting
6096fb1427
update puppeteer plugin to create a PuppeteerBinProvider for installing browsers 2024-09-21 04:12:34 -07:00
Nick Sweeting
6c39d27ccb
update singlefile plugin to use new npm binprovider and support installing 2024-09-21 04:12:33 -07:00
Nick Sweeting
dd6d7e4975
fix npm and pip binprovider setup and paths search 2024-09-21 04:12:33 -07:00
Nick Sweeting
30def925e7
move all ansible files into plugantic folder for now 2024-09-21 04:12:33 -07:00
Nick Sweeting
11f369ee2d
bump subdependency versions 2024-09-21 04:12:33 -07:00
Nick Sweeting
61df9ea059
fix duplicate when 2024-09-17 02:04:41 -07:00
Nick Sweeting
2c8779736a
change default node version to 21 2024-09-17 02:03:28 -07:00
Nick Sweeting
f29f72f383
fix os checking for npm install 2024-09-17 01:57:06 -07:00
Nick Sweeting
a5cefb5464
install nodesource first 2024-09-17 01:46:02 -07:00
Nick Sweeting
19c7b9c24e
install nodejs and npm packages properly in npm ansible 2024-09-17 01:42:06 -07:00
Nick Sweeting
7ab1a0b873
fix singlefile and puppeteer ansible install 2024-09-17 01:33:32 -07:00
Nick Sweeting
5c0aa6fe59
more ansible fixes 2024-09-17 01:12:49 -07:00
Nick Sweeting
c55cd46ecb
consolidate ansible setup into roles dir 2024-09-17 00:48:47 -07:00
Nick Sweeting
25db6826ec
ignore lib dirs 2024-09-17 00:47:55 -07:00
Nick Sweeting
8d69469887
silence ansible errors about implicit localhost 2024-09-15 20:31:11 -07:00
Nick Sweeting
fab80632b7
add ansible runner code to get facts after execution and benedict 2024-09-15 20:29:02 -07:00
Nick Sweeting
e9ddac0219
fix ansible installed_packages and cacheable facts 2024-09-15 20:28:35 -07:00
Nick Sweeting
56b851ea1b
more ansible playbooks improvements 2024-09-13 04:55:40 -07:00
Nick Sweeting
8557e77a70
add ansible playbooks 2024-09-13 03:27:38 -07:00
Nick Sweeting
3bbf8f69ab
cleanup settings.py sqlite settings more 2024-09-13 03:27:38 -07:00
Nick Sweeting
c887af0278
minor ruff fixes 2024-09-12 02:00:07 -07:00
Nick Sweeting
c00afce71f
upgrade dependency versions to django 5.1 minimum 2024-09-11 17:08:10 -07:00
Nick Sweeting
eae11cba19
add recommended SQLite db connection settings to avoid single-writer lock contention 2024-09-11 16:50:44 -07:00
Nick Sweeting
0187c8b6cb
bump version to 0.8.4 2024-09-10 03:10:30 -07:00
Nick Sweeting
a13f71a86c
allow supervisord to start if pid file is stale 2024-09-10 03:10:10 -07:00
Nick Sweeting
cecca8d169
allow deleting results from list page 2024-09-10 03:09:43 -07:00
Nick Sweeting
f5c878b267
point select2 js resources to local statifiles 2024-09-10 01:51:08 -07:00
Nick Sweeting
0640018426
bump packages 2024-09-10 01:50:49 -07:00
Nick Sweeting
0bd678c30f
fix init 2024-09-10 00:37:01 -07:00
Nick Sweeting
d680c48942
avoid auto-starting all supervisord workers on startup 2024-09-10 00:19:32 -07:00
Nick Sweeting
8d1d5df43a
add proxy model for SinglefileResult wrapping ArchiveResult 2024-09-10 00:05:46 -07:00
Nick Sweeting
04cb6bdfd6
ignore healtcheck requests in daphne logs 2024-09-10 00:05:46 -07:00
Nick Sweeting
1ce09b88d7
show logs and workers in Django Admin data views 2024-09-10 00:05:45 -07:00
Nick Sweeting
60154fba5f
add django_huey, huey_monitor, and replace Threads with huey tasks 2024-09-10 00:05:45 -07:00
Nick Sweeting
4df90fbb40
change plugins to have both a .register that runs at import and .ready that runs later 2024-09-10 00:05:45 -07:00
Nick Sweeting
f1cca5bbba
ignore tmp dirs 2024-09-10 00:05:45 -07:00
Nick Sweeting
db5428a486
increase maximum form field POST param size 2024-09-10 00:05:45 -07:00
Nick Sweeting
31576e21a2
fix plugins and modules dataviews 2024-09-06 06:14:18 -07:00
Nick Sweeting
3f76e0a87f
fix migrations import errors 2024-09-06 03:48:52 -07:00
Nick Sweeting
ab517d9848
fix infinite recursion on pydantic instantiation 2024-09-06 03:19:03 -07:00
Nick Sweeting
cfe5798ade
bump version to 0.8.3 2024-09-06 02:59:14 -07:00
Nick Sweeting
6e2ddc14df
make archivebox server spawn daphne process instead of runserver 2024-09-06 02:55:06 -07:00
Nick Sweeting
f6ca48835a
hide more 404 and 304 errors from daphne logs 2024-09-06 02:54:53 -07:00
Nick Sweeting
52386d9c16
run all blocking commands in background threads and show nice UI messages as confirmation 2024-09-06 02:54:22 -07:00
Nick Sweeting
b56b1cac35
cleanup plugantic and pkg apps, make BaseHook actually create its own settings 2024-09-06 01:48:18 -07:00
Nick Sweeting
0e79a8b683
minor type hint fixes 2024-09-05 23:24:44 -07:00
Nick Sweeting
2e1e1945f2
add django-object-actions to provide Regenerate ABID button 2024-09-05 23:19:21 -07:00
Nick Sweeting
00aa7dc19f
setup daphne and django channels to replace runserver 2024-09-05 21:45:43 -07:00
Nick Sweeting
a190745f36
add rich support for logging and ruff for formatting 2024-09-05 21:44:18 -07:00
Nick Sweeting
2c48dabfab
add preliminary support for ASGI/daphne serving 2024-09-05 21:43:42 -07:00
Nick Sweeting
26291f81ef
prevent ABID drift entirely when its set to False on a model 2024-09-05 21:43:14 -07:00
Nick Sweeting
c76c50e71f
add HTTP byte range request support to media file serving 2024-09-05 21:41:49 -07:00
Nick Sweeting
ba6c1fd69b
minor formatting and fixes 2024-09-05 04:39:46 -07:00
Nick Sweeting
d50aed9185
improve new Hook system docstring 2024-09-05 03:51:54 -07:00
Nick Sweeting
44669fab73
add BaseHook concept to underlie all Plugin hooks 2024-09-05 03:36:18 -07:00
Nick Sweeting
ed5357cec9
add migrations for datetime field renames 2024-09-04 23:44:13 -07:00
Nick Sweeting
0fef2357c7
change default datetime template rendering to include seconds 2024-09-04 23:43:25 -07:00
Nick Sweeting
0c236b034b
allow accessing admin change pages by .abid as well as .id 2024-09-04 23:43:07 -07:00
Nick Sweeting
cbf2a8fdc3
rename datetime fields to _at, massively improve ABID generation safety and determinism 2024-09-04 23:42:36 -07:00
Nick Sweeting
68a39b7392
remove .old_id entirely and make ABID generation only happen once on initial save 2024-09-04 16:40:15 -07:00
Nick Sweeting
22a6666bab
fix pkg management subcommand 2024-09-04 02:37:19 -07:00
Nick Sweeting
9072f4a999
fix ABID added to default to bookmarked value 2024-09-04 02:34:27 -07:00
Nick Sweeting
4427869ae8
fix ABID generation by chopping ts_src precision to consistent length 2024-09-04 02:02:29 -07:00
Nick Sweeting
d060eaa499
abid gradual improvements, some regrets 2024-09-04 00:08:14 -07:00
Nick Sweeting
ae13f1811f
better ABID display in admin UI 2024-09-03 17:11:10 -07:00
Nick Sweeting
3d7dd3c9cf
bump pydantic-pkgr 2024-09-03 14:17:50 -07:00
Nick Sweeting
f915ef6c5e
try to improve django console log 404 and 200 error filtering 2024-09-03 14:17:02 -07:00
Nick Sweeting
01094ecb03
fix REST API CSRF and auth handling 2024-09-03 14:16:44 -07:00
Nick Sweeting
41a318a8bd
fix config loading precedence order 2024-09-03 03:22:37 -07:00
Nick Sweeting
8d4ace017d
fix Binary overrides getting wiped after init 2024-09-03 02:50:30 -07:00
Nick Sweeting
a18857dbb9
add ytldp extractor stub 2024-09-03 01:40:38 -07:00
Nick Sweeting
b927f5f2a9
add FFMPEG_BINARY to ytdlp plugin 2024-09-03 01:39:02 -07:00
Nick Sweeting
c1c55d6da7
fix API token_auth and CSRF setup 2024-09-03 01:21:13 -07:00
Nick Sweeting
9af260df16
BasePlugin system expanded and registration system improved 2024-09-03 00:58:50 -07:00
Nick Sweeting
f1579bfdcd
add NODE_BIN_PATH to dynamic CONFIG and make ANSI an AttrDict 2024-09-03 00:19:18 -07:00
Nick Sweeting
23c7d2f860
bump pydantic-pkgr 2024-09-02 15:06:10 -07:00
Nick Sweeting
3ea116f0b6
tweak Pkgs app name and admin button CSS 2024-09-02 01:14:41 -07:00
Nick Sweeting
9be7e1a0bc
change Snapshot admin action UI button text 2024-09-02 01:14:06 -07:00
Nick Sweeting
5f7bca298c
bump USER_AGENT config default to chrome v128 2024-09-02 01:13:50 -07:00
Nick Sweeting
dd05ad04fa
disable cookie auth in API because csrf=False 2024-09-02 01:13:19 -07:00
Nick Sweeting
da76a84c45
rename pkgs app to pkg 2024-08-30 03:32:36 -07:00
Nick Sweeting
6e13cd4820
fix loading of plugin dependencies and bump pydantic_pkgr version 2024-08-30 03:30:28 -07:00
Nick Sweeting
a8f00caff8
bump singlefile version 2024-08-29 22:33:47 -07:00
Nick Sweeting
57ed2ebcb2
switch youtube_dl install for yt-dlp 2024-08-28 03:46:45 -07:00
Nick Sweeting
038e9f32af
bump deps versions 2024-08-28 03:03:09 -07:00
Nick Sweeting
4ac980a832
hide binary loading debug output for now 2024-08-28 03:02:54 -07:00
Nick Sweeting
1e73a06ba0
change ABIDModel.created to use AutoTimeField seeded on .save instead of auto_now_add so that ts_src for ABID is available on creation before DB row is created 2024-08-28 03:02:37 -07:00
Nick Sweeting
7a734ce410
fix archivebox setup User model import 2024-08-28 03:01:35 -07:00
Nick Sweeting
e558d71b10
fix uuid checks on save 2024-08-28 03:01:19 -07:00
Nick Sweeting
6456cb1727
fix NOT NULL constraint failed: core_snapshot.created_by_id 2024-08-28 00:51:16 -07:00
Nick Sweeting
b789be098d
fix type hint 2024-08-28 00:26:37 -07:00
Nick Sweeting
d0fefc0279
add chunk_size=500 to more iterator calls 2024-08-27 19:28:00 -07:00
Nick Sweeting
44849e1ba2
fix python vendor dependendcies 2024-08-26 20:24:25 -07:00
Nick Sweeting
d7e333b5cf
fix settings.py for prod 2024-08-26 20:20:33 -07:00
Nick Sweeting
24fe958ff3
massively improve Snapshot admin list view query performance 2024-08-26 20:16:43 -07:00
Nick Sweeting
6c4f3fc83a
fix chrome headless=new arg 2024-08-26 20:15:36 -07:00
Nick Sweeting
6ffa710bb3
fix headers Elapsed timedelta is not a JSON-serializable 2024-08-26 20:15:22 -07:00
Nick Sweeting
917af4b1d7
add pydantic logfire and improve settings for debugging 2024-08-26 20:14:47 -07:00
Nick Sweeting
3a57c28ca8
bump pydantic_pkgr version to 0.1.4 2024-08-23 02:03:26 -07:00
Nick Sweeting
5fe3edd79a
new plugin loading system 2024-08-23 02:02:34 -07:00
Nick Sweeting
34389e5e7c
improve CSRF_TRUSTED_ORIGINS loading logic 2024-08-23 02:01:40 -07:00
Nick Sweeting
c6a80ab256
use pydantic_pkgr to parse semver versions 2024-08-23 02:01:24 -07:00
Nick Sweeting
2c2d034d6d
move to new vendoring fallback logic 2024-08-23 02:01:02 -07:00
Nick Sweeting
9c35f3ddb7
add new CSRF_TRUSTED_ORIGINS config option 2024-08-22 18:40:47 -07:00
Nick Sweeting
1a03db2b1d
fix api auth endpoint checking wrong user field 2024-08-22 18:28:23 -07:00
Nick Sweeting
73a3e6aad0
handle tag with no slug or name 2024-08-22 18:25:15 -07:00
Nick Sweeting
cec8015b5b
fix missing STDOUT 2024-08-22 17:58:58 -07:00
Nick Sweeting
38ca5c3228
add extra info to headers.json 2024-08-22 17:57:40 -07:00
Nick Sweeting
80d7a30f13
improve version_str detection 2024-08-22 17:57:33 -07:00
Nick Sweeting
1d31b88fa3
fix migration failing when Tag name is empty 2024-08-22 16:30:25 -07:00
Nick Sweeting
9ecc2ff91c
bump more deps 2024-08-22 15:36:11 -07:00
Nick Sweeting
09553d8340
hardcode EXTRACTOR_CHOICES to prevent nondeterministic migrations 2024-08-22 15:36:02 -07:00
Nick Sweeting
0a5b22700c
bump deps 2024-08-22 15:20:52 -07:00
Nick Sweeting
afe1307617
fix created_by field migration to create User properly if none exists 2024-08-22 15:20:36 -07:00
Nick Sweeting
9b1659c72f
make created_by_id autoapply to any ArchiveResults created under Snapshot 2024-08-20 19:43:07 -07:00
Nick Sweeting
c30ae1d2cb
add created_by_id to all Snapshot creation functions 2024-08-20 19:28:28 -07:00
Nick Sweeting
aa282daadf
show CONSTANTS in separate section of django admin config 2024-08-20 19:27:21 -07:00
Nick Sweeting
ea81f2fc14
use ABIDAdmin as base ModelAdmin 2024-08-20 19:26:40 -07:00
Nick Sweeting
57e48b432c
show redacted API tokens in django admin 2024-08-20 19:25:16 -07:00
Nick Sweeting
ff619fd31f
add api migrations 2024-08-20 18:31:40 -07:00
Nick Sweeting
0285aa52a0
config and attr access improvements 2024-08-20 18:31:21 -07:00
Nick Sweeting
4ae186dfca
fix ABID generation consistency when self._state.adding is True 2024-08-20 05:56:19 -07:00
Nick Sweeting
9d2116ad9a
migration to make fields non editable 2024-08-20 03:46:04 -07:00
Nick Sweeting
9c8ece4d7c
minor css and APIToken Admin UI fixes 2024-08-20 03:45:49 -07:00
Nick Sweeting
75018ed10b
Merge branch 'dev' into feat/search-fields 2024-08-20 03:37:56 -07:00
Nick Sweeting
05880dba8b
bump version to 0.8.2 2024-08-20 03:36:07 -07:00
Nick Sweeting
267964881c
fix tags editor in Snapshot list view 2024-08-20 03:29:57 -07:00
Nick Sweeting
849b4963a1
add migrations 2024-08-20 01:58:44 -07:00
Nick Sweeting
52a813aa80
fix title display in admin UI and abid filter matching in urls 2024-08-20 01:58:36 -07:00
Nick Sweeting
9273db528e
fix abid generation migrations to be historically consistent 2024-08-20 01:58:19 -07:00
Nick Sweeting
506b3d28d4
fix admin UI TagInline and ArchiveResultInline form POST handling 2024-08-20 01:57:07 -07:00
Nick Sweeting
54acfd9f86
improve REST API filter parameters and pagination 2024-08-20 01:56:37 -07:00
Nick Sweeting
850448b42c
add salt args in more places 2024-08-20 01:56:07 -07:00
Nick Sweeting
c4ef2993b2
update REST API and Admin UI to use new id and old_id exclusively 2024-08-19 20:00:04 -07:00
Nick Sweeting
344e902fc6
migrate SnapshotTag to use new snapshot id 2024-08-19 19:42:25 -07:00
Nick Sweeting
74c11d41be
dont allow modifying Snapshot.timestamp as it would break folder dirs 2024-08-19 19:42:01 -07:00
Nick Sweeting
c68a66f74e
add salt kwarg to abid generation funcs 2024-08-19 19:41:43 -07:00
Nick Sweeting
a49739b41c
add hacky workaround for TagInline not showing in admin Snapshot change view 2024-08-19 18:36:35 -07:00
Nick Sweeting
cf2faecf61
add migrations for SnapshotTag through model 2024-08-19 18:36:20 -07:00
Nick Sweeting
7164fb961c
hotlink to snapshots using generic search query link instead of id__startswith 2024-08-19 18:35:35 -07:00
Nick Sweeting
3148d2a3ef
add squashmigrations to allowed mgmgt command list 2024-08-19 18:35:07 -07:00
Nick Sweeting
2c157f0b5b
increase max POST field limit for admin 2024-08-19 18:34:52 -07:00
Nick Sweeting
18ebaed7ed
fix admin UI display of tags 2024-08-19 18:34:01 -07:00
Nick Sweeting
57d31b2b14
fix snapshot uuid 2024-08-18 01:07:21 -07:00
Nick Sweeting
8c50257fe9
move snapshot id to old_id 2024-08-18 00:24:38 -07:00
Nick Sweeting
951025228f
add tag endpoint 2024-08-18 00:24:14 -07:00
Nick Sweeting
a271bcb4ce
use new ids for ArchiveResult API 2024-08-17 23:03:43 -07:00
Nick Sweeting
f72debfdb2
migrate ArchiveResult.id to old_id, and make uuid main id 2024-08-17 22:58:35 -07:00
Nick Sweeting
033ec08d0c
save snapshot ids during migration 2024-08-17 21:56:45 -07:00
Nick Sweeting
4d0bbfccfc
add uri salt and fix api url namespaces 2024-08-17 21:56:23 -07:00
Nick Sweeting
fba3995d86
reorder version output 2024-08-17 19:38:51 -07:00
Nick Sweeting
6a33de6715
remove archive.today link and add JSON 2024-08-17 19:31:43 -07:00
Nick Sweeting
3a87a7fb8c
re-arrange snapshot detail page preference and add opus support 2024-08-17 19:31:34 -07:00
Nick Sweeting
5e0cc926f1
show ulid in archivebox admin 2024-08-17 19:31:18 -07:00
Nick Sweeting
b7745ae37a
add uri salt when creating ABID 2024-08-17 19:30:58 -07:00
Lucas Schwiderski
10f0c4a077
Implement searching individual fields 
So far only for the public view, since we already have a custom search
form there, where this is easy to add.
This initial implementation supports the common set of metadata fields
that the searchbar placeholder also mentions, but adding more fields
is trivial.
 2024-06-19 16:50:16 +02:00
Nick Sweeting
99e6f0c93f
bump versions 2024-06-04 04:17:33 -07:00
Nick Sweeting
de489d3c60
minor snapshot details ui fixes and migrations log msg improvements 2024-06-04 04:17:32 -07:00
Nick Sweeting
f574d34357
wrap migrations maker in try catch 2024-06-03 03:02:00 -07:00
Nick Sweeting
c63917a22d
fix more CHROME_USER_DATA_DIR issues 2024-06-03 02:51:11 -07:00
Nick Sweeting
1cd62ecc61
add filesizes and stray files in snapshot dir to snapshot_live ui 2024-06-03 02:31:35 -07:00
Nick Sweeting
729f05ab93
fix snapshot_live iframe scrollTo not working due to origin sandbox rules 2024-06-02 19:15:12 -07:00
Nick Sweeting
11a24d9640
merge migrations between branches 2024-06-02 19:14:58 -07:00
Nick Sweeting
b3f2a71c93
bump requirements and version to 0.8.1 2024-06-02 19:14:45 -07:00
Nick Sweeting
07d80db5aa
fix CHROME_USER_DATA_DIR split str error 2024-06-02 19:14:20 -07:00
Nick Sweeting
f72bae8eec
autodiscover plugins on startup 2024-05-17 20:48:11 -07:00
Nick Sweeting
48becde9b4
add new pydantic-based plugin system 2024-05-17 20:13:54 -07:00
Nick Sweeting
e4176dbf7a
change live snapshot preview iframe sandbox rules 2024-05-17 20:12:37 -07:00
Nick Sweeting
774ce3fda7
fix singlefile extractor exception when result is none 2024-05-17 20:12:18 -07:00
Nick Sweeting
29c7aa26bc
show original section titles in config admin ui 2024-05-17 20:11:54 -07:00
Nick Sweeting
acfd346440
make abids searchable in the admin ui 2024-05-17 20:11:31 -07:00
Nick Sweeting
a1afd0211f
fix abid calculation 2024-05-17 20:11:00 -07:00
Nick Sweeting
fdf6f465db
add docstrings 2024-05-14 18:46:38 -07:00
Nick Sweeting
406f57031a
add API support for obj.pk .uuid .abid 2024-05-13 08:52:02 -07:00
Nick Sweeting
a4cc10d7f8
add migrations for third round of field changes 2024-05-13 07:50:22 -07:00
Nick Sweeting
241a7c6ab2
add created, modified, updated, created_by and update django admin 2024-05-13 07:50:07 -07:00
Nick Sweeting
1ba8215072
only use domain part of uri for hash 2024-05-13 07:49:36 -07:00
Nick Sweeting
206e7e74b3
add migrations to create and populate ABIDField and UUIDField values 2024-05-13 05:13:42 -07:00
Nick Sweeting
0420662174
switch everywhere to use Snapshot.pk and ArchiveResult.pk instead of id 2024-05-13 05:12:12 -07:00
Nick Sweeting
9733b8d04c
remove accidentally commited db 2024-05-13 02:38:02 -07:00
Nick Sweeting
4f9f22e024
create abid_utils with new ABID type for ArchiveBox IDs 2024-05-13 02:37:48 -07:00
Nick Sweeting
f896e5dbeb
switch from monkey patching WebhookModel to using swappable 2024-05-13 02:36:15 -07:00
Nick Sweeting
e97d779cd3
move monkey patches to dedicated file 2024-05-13 02:35:19 -07:00
Nick Sweeting
b5ad134264
dont wait for ipython history saver thread before shell exit 2024-05-12 19:25:55 -07:00
Nick Sweeting
ce833e8ead
automatically create storage directories and symlinks based on ulid 2024-05-12 05:21:58 -07:00
Nick Sweeting
33bc4622a0
add ulid and typeid to Snapshot and ArchiveResult 2024-05-12 04:45:34 -07:00
Nick Sweeting
0529099639
create cache dir and sqlite db 2024-05-12 01:42:34 -07:00
Nick Sweeting
e1a04729b3
wait for bg threads to finish before exiting 2024-05-12 01:42:20 -07:00
Nick Sweeting
0438924491
remove unused template vars 2024-05-11 22:43:26 -07:00
Nick Sweeting
2b4b6e5b3a
select single snapshot row automatically when hotlinking to admin actions 2024-05-11 22:33:39 -07:00
Nick Sweeting
27809f2976
fix python3.10 nested fstring quotes 2024-05-11 22:33:27 -07:00
Nick Sweeting
baf24d2d6a
change wording on 404 snapshot detail page 2024-05-11 22:33:17 -07:00
Nick Sweeting
8841e8b181
add new live django template for snapshot detail page 2024-05-11 22:33:02 -07:00
Nick Sweeting
457c42bf84
load EXTRACTORS dynamically using importlib.import_module 2024-05-11 22:28:59 -07:00
Nick Sweeting
c7f55fc3ba
make CHROME_USER_DATA_DIR error quieter 2024-05-11 22:16:10 -07:00
Nick Sweeting
102e87578c
re-lock packages for python3.10 because we have to support it still 2024-05-11 15:06:19 -07:00
Nick Sweeting
913590ee39
explain weird use of ellipses magic value 2024-05-11 15:02:43 -07:00
Nick Sweeting
b90ba6c909
change header auth to use X-ArchiveBox-API-Key so it doesnt conflict with other auth headers 2024-05-08 20:02:47 -07:00
Nick Sweeting
4c5a3fba8b
more fixes for wget_output_path 2024-05-07 05:38:29 -07:00
Nick Sweeting
f2729c9dc7
revert wget character limit back to windows, ascii is allows MORE not less 2024-05-07 05:24:16 -07:00
Nick Sweeting
9b21ce490e
add workaround logic to catch paths that are too long or contain unprintable characters 2024-05-07 05:03:23 -07:00
Nick Sweeting
f62cb5fb43
change wget to use stricter ascii filepath normalization 2024-05-07 05:03:01 -07:00
Nick Sweeting
f770bba3cf
fix OSError 36 caused by checking for path that is too long to exist 2024-05-07 04:12:07 -07:00
Nick Sweeting
ef856e8051
Merge branch 'dev' into issue1316 2024-05-06 23:14:16 -07:00
Nick Sweeting
27d5d1ddc8
revert queryset intersection back to union for search results 2024-05-06 23:13:52 -07:00
Nick Sweeting
f472705d10
Change phrasing (#1419) 2024-05-06 23:11:26 -07:00
Nick Sweeting
3095265880
fix inner quote not escaped 2024-05-06 23:10:59 -07:00
Brandl
8ccd606973
Fix quotation 
Fixes:
=> ERROR [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt                                       1.7s 
------                                                                                                                                      
 > [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt:                                                  
1.665 Traceback (most recent call last):                                                                                                    
1.665   File "/usr/local/bin/archivebox", line 5, in <module>                                                                               
1.665     from archivebox.cli import main                                                                                                   
1.665   File "/app/archivebox/cli/__init__.py", line 83, in <module>
1.665     SUBCOMMANDS = list_subcommands()
1.665                   ^^^^^^^^^^^^^^^^^^
1.665   File "/app/archivebox/cli/__init__.py", line 43, in list_subcommands
1.665     module = import_module('.archivebox_{}'.format(subcommand), __package__)
1.665              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.665   File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
1.665     return _bootstrap._gcd_import(name[level:], package, level)
1.665            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.666   File "/app/archivebox/cli/archivebox_add.py", line 11, in <module>
1.666     from ..main import add
1.666   File "/app/archivebox/main.py", line 233
1.666     f'COMMIT_HASH={COMMIT_HASH[:7] if COMMIT_HASH else 'unknown'}',
1.666                                                         ^^^^^^^
1.666 SyntaxError: f-string: expecting '}'
 2024-05-06 21:04:14 +02:00
Nick Sweeting
027c029316
redact passwords, keys, and secret tokens in admin UI 2024-05-06 11:06:42 -07:00
Nick Sweeting
8667ed29f1
improve API webhooks helptext and change app_label to API 2024-05-06 08:11:01 -07:00
Evan Boehs
f998647350
change phrasing 2024-05-06 10:32:36 -04:00
Nick Sweeting
d782bafe2e
fix storages missing stackfiles error 2024-05-06 07:14:01 -07:00
Nick Sweeting
47666ec26b
show webhooks config in django admin 2024-05-06 07:13:54 -07:00
Nick Sweeting
f067451267
fix django timezone.utc removed in 5.0 2024-05-06 07:13:25 -07:00
Nick Sweeting
c7fc9c004f
add django-signal-webhooks 2024-05-06 06:58:03 -07:00
Nick Sweeting
3805a1730d
add 0002 api migration 2024-04-30 21:45:02 -07:00
Nick Sweeting
e20eb52f15
fix COMMIT_HASH missing error 2024-04-30 21:43:22 -07:00
Nick Sweeting
18a5b6e99c
move EXTERNAL_LOCATIONS to data locations in version output 2024-04-25 21:36:43 -07:00
Nick Sweeting
6a6ae7468e
fix lint errors 2024-04-25 21:36:11 -07:00
Nick Sweeting
4a5ad32040
add django-requests-tracker 2024-04-25 18:02:01 -07:00
Nick Sweeting
af669d2f37
rename api files for clarity 2024-04-25 05:55:47 -07:00
Nick Sweeting
75153252dc
big overhaul of REST API, split into auth, core, and cli methods 2024-04-25 03:56:22 -07:00
Nick Sweeting
e5aba0dc2e
fix urljoin bug causing multiple slashes to be merged into one 2024-04-24 19:41:31 -07:00
Nick Sweeting
6cb357e76c
fix fix_url_from_markdown assertion to be valid url 2024-04-24 19:41:11 -07:00
Nick Sweeting
128419f991
expand comment about markdown url trailing paren trimming 2024-04-24 17:50:18 -07:00
Nick Sweeting
beb3932d80
replace uses of URL_REGEX with find_all_urls to handle markdown better 2024-04-24 17:45:45 -07:00
Nick Sweeting
c6d644be29
Merge branch 'dev' into issue1316 2024-04-24 16:24:16 -07:00
Nick Sweeting
98c5e69203
bump lockfiles 2024-04-24 14:38:21 -07:00
Nick Sweeting
8dcfa93ec6
Merge branch 'main' into dev 2024-04-24 14:32:07 -07:00
Nick Sweeting
17f40f3ada
Merge branch 'dev' into fix-URL_REGEX 2024-04-23 19:53:58 -07:00
Nick Sweeting
c6f8a33a63
Update util.py 2024-04-23 19:53:18 -07:00
Nick Sweeting
63fc317229
minor pylint fixes in logging_util 2024-04-23 17:45:36 -07:00
Nick Sweeting
756e159dfe
add new bin/lock_pkgs.sh to generate package lockfiles consistently 2024-04-23 17:45:36 -07:00
longzai
e4dc2701ef fix URL_REGEX 2 2024-04-11 15:51:55 +08:00
Brandl
5f9aac18f2
api v1 2024-04-10 01:29:24 +02:00
longzai
4ae765ec27 fix the URL_REGEX used in generic_html parsers 
Signed-off-by: longzai <437172242@qq.com>
 2024-04-08 04:53:05 +08:00
Nick Sweeting
e48159b8a0 cleanup docker-compose by storing crontabs in data dir 2024-03-26 15:24:05 -07:00
Nick Sweeting
ac73fb5129 merge fixes 2024-03-26 15:22:40 -07:00
Nick Sweeting
b4c3aa5097 Merge branch 'main' into dev 2024-03-26 15:01:36 -07:00
Nick Sweeting
a4453b6f87
fix PERSONAS PERSONAS_DIR typo 2024-03-26 14:19:25 -07:00
jim winstead
8b1b01e508 Update to Django 4.2.x, now in LTS until April 2026 2024-03-25 17:46:01 -07:00
Nick Sweeting
c0b5dbcecb create new data/personas dir to hold cookies and chrome profiles 2024-03-18 14:41:39 -07:00
Nick Sweeting
c5bb99dce1 explicitly use Default profile inside user data dir 2024-03-18 14:40:40 -07:00
Nick Sweeting
1fc5d7c5c8 add USER_AGENT config option to set all USER_AGENTs at once 2024-03-18 14:39:09 -07:00
jim winstead
5478d13d52 Add generic_jsonl parser 
Resolves #1369
 2024-03-14 15:42:29 -07:00
Nick Sweeting
ca2c484a8e
Add _EXTRA_ARGS for various extractors (#1360) 2024-03-14 01:55:09 -07:00
Nick Sweeting
48f4b12ae2
Use COOKIES_FILE to fetch page titles (#1364) 2024-03-14 01:52:44 -07:00
Nick Sweeting
099f7d00fe
Use feedparser for RSS parsing (#1362) 
Fixes #1171
Fixes #870 (probably, would need to test against a Wallabag Atom file to
Fixes #135
Fixes #123
Fixes #106
 2024-03-14 01:51:45 -07:00
Nick Sweeting
3512dc7e60
Disable searching for existing chrome user profiles by default 2024-03-14 00:58:45 -07:00
Ben Muthalaly
f4deb97f59 Add ARGS and EXTRA_ARGS for Mercury extractor 2024-03-05 21:15:38 -06:00
Ben Muthalaly
d8cf09c21e Remove unnecessary variable length args for dedupe 2024-03-05 21:13:45 -06:00
Ben Muthalaly
5082d61613 Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file 2024-03-05 02:03:03 -06:00
Ben Muthalaly
4686da91e6 Fix cookies being set incorrectly 2024-03-05 01:48:35 -06:00
Naomi Phillips
a729480b75
Add COOKIES_FILE support for singlefile extractor 2024-03-03 02:32:46 -05:00
Ben Muthalaly
d74ddd42ae Flip dedupe precedence order 2024-03-01 14:50:32 -06:00
jim winstead
9f462a87a8 Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers 
The feedparser packages has 20 years of history and is very good at parsing
RSS and Atom, so use that instead of ad-hoc regex and XML parsing.

The medium_rss and shaarli_rss parsers weren't touched because they are
probably unnecessary. (The special parse for pinboard is just needing because
of how tags work.)

Doesn't include tests because I haven't figured out how to run them in the
docker development setup.

Fixes #1171
 2024-03-01 11:25:45 -08:00
Nick Sweeting
a577d1ed23
Merge branch 'dev' into title-cookies-file 2024-02-29 21:29:36 -08:00
Nick Sweeting
fe11e1c2f4
check if COOKIE_FILE is file 2024-02-28 18:19:44 -08:00
jim winstead
ccabda4c7d Handle list of tags in JSON, and be more clever about comma vs. space 2024-02-28 17:38:49 -08:00
jim winstead
178e676e0f Fix JSON parser by not always mangling the input 
Rather than by assuming the JSON file we are parsing has junk at the beginning
(which maybe only used to happen?), try parsing it as-is first, and then fall
back to trying again after skipping the first line

Fixes #1347
 2024-02-27 14:48:19 -08:00
Ben Muthalaly
68326a60ee Add cookies file to http request in download_url 2024-02-27 15:30:31 -06:00
Ben Muthalaly
4d9c5a7b4b Add CHROME_EXTRA_ARGS 
Also fix `YOUTUBEDL_EXTRA_ARGS`.
 2024-02-23 18:40:03 -06:00
Ben Muthalaly
ab8f395e0a Add YOUTUBEDL_EXTRA_ARGS 2024-02-23 15:40:31 -06:00
Ben Muthalaly
4e69d2c9e1 Add EXTRA_*_ARGS for wget, curl, and singlefile 2024-02-22 23:04:11 -06:00
Nick Sweeting
8b9bc3dec8 minor fixes 2024-02-22 04:50:22 -08:00
Nick Sweeting
6a4e568d1b new archivebox update speed improvements 2024-02-22 04:50:22 -08:00