1796 commits

Author	SHA1	Message	Date
Nick Sweeting	0e79a8b683	minor type hint fixes	2024-09-05 23:24:44 -07:00
Nick Sweeting	2e1e1945f2	add django-object-actions to provide Regenerate ABID button	2024-09-05 23:19:21 -07:00
Nick Sweeting	00aa7dc19f	setup daphne and django channels to replace runserver	2024-09-05 21:45:43 -07:00
Nick Sweeting	a190745f36	add rich support for logging and ruff for formatting	2024-09-05 21:44:18 -07:00
Nick Sweeting	2c48dabfab	add preliminary support for ASGI/daphne serving	2024-09-05 21:43:42 -07:00
Nick Sweeting	26291f81ef	prevent ABID drift entirely when its set to False on a model	2024-09-05 21:43:14 -07:00
Nick Sweeting	c76c50e71f	add HTTP byte range request support to media file serving	2024-09-05 21:41:49 -07:00
Nick Sweeting	ba6c1fd69b	minor formatting and fixes	2024-09-05 04:39:46 -07:00
Nick Sweeting	d50aed9185	improve new Hook system docstring	2024-09-05 03:51:54 -07:00
Nick Sweeting	44669fab73	add BaseHook concept to underlie all Plugin hooks	2024-09-05 03:36:18 -07:00
Nick Sweeting	ed5357cec9	add migrations for datetime field renames	2024-09-04 23:44:13 -07:00
Nick Sweeting	0fef2357c7	change default datetime template rendering to include seconds	2024-09-04 23:43:25 -07:00
Nick Sweeting	0c236b034b	allow accessing admin change pages by .abid as well as .id	2024-09-04 23:43:07 -07:00
Nick Sweeting	cbf2a8fdc3	rename datetime fields to _at, massively improve ABID generation safety and determinism	2024-09-04 23:42:36 -07:00
Nick Sweeting	68a39b7392	remove .old_id entirely and make ABID generation only happen once on initial save	2024-09-04 16:40:15 -07:00
Nick Sweeting	22a6666bab	fix pkg management subcommand	2024-09-04 02:37:19 -07:00
Nick Sweeting	9072f4a999	fix ABID added to default to bookmarked value	2024-09-04 02:34:27 -07:00
Nick Sweeting	4427869ae8	fix ABID generation by chopping ts_src precision to consistent length	2024-09-04 02:02:29 -07:00
Nick Sweeting	d060eaa499	abid gradual improvements, some regrets	2024-09-04 00:08:14 -07:00
Nick Sweeting	ae13f1811f	better ABID display in admin UI	2024-09-03 17:11:10 -07:00
Nick Sweeting	3d7dd3c9cf	bump pydantic-pkgr	2024-09-03 14:17:50 -07:00
Nick Sweeting	f915ef6c5e	try to improve django console log 404 and 200 error filtering	2024-09-03 14:17:02 -07:00
Nick Sweeting	01094ecb03	fix REST API CSRF and auth handling	2024-09-03 14:16:44 -07:00
Nick Sweeting	41a318a8bd	fix config loading precedence order	2024-09-03 03:22:37 -07:00
Nick Sweeting	8d4ace017d	fix Binary overrides getting wiped after init	2024-09-03 02:50:30 -07:00
Nick Sweeting	a18857dbb9	add ytldp extractor stub	2024-09-03 01:40:38 -07:00
Nick Sweeting	b927f5f2a9	add FFMPEG_BINARY to ytdlp plugin	2024-09-03 01:39:02 -07:00
Nick Sweeting	c1c55d6da7	fix API token_auth and CSRF setup	2024-09-03 01:21:13 -07:00
Nick Sweeting	9af260df16	BasePlugin system expanded and registration system improved	2024-09-03 00:58:50 -07:00
Nick Sweeting	f1579bfdcd	add NODE_BIN_PATH to dynamic CONFIG and make ANSI an AttrDict	2024-09-03 00:19:18 -07:00
Nick Sweeting	23c7d2f860	bump pydantic-pkgr	2024-09-02 15:06:10 -07:00
Nick Sweeting	3ea116f0b6	tweak Pkgs app name and admin button CSS	2024-09-02 01:14:41 -07:00
Nick Sweeting	9be7e1a0bc	change Snapshot admin action UI button text	2024-09-02 01:14:06 -07:00
Nick Sweeting	5f7bca298c	bump USER_AGENT config default to chrome v128	2024-09-02 01:13:50 -07:00
Nick Sweeting	dd05ad04fa	disable cookie auth in API because csrf=False	2024-09-02 01:13:19 -07:00
Nick Sweeting	da76a84c45	rename pkgs app to pkg	2024-08-30 03:32:36 -07:00
Nick Sweeting	6e13cd4820	fix loading of plugin dependencies and bump pydantic_pkgr version	2024-08-30 03:30:28 -07:00
Nick Sweeting	a8f00caff8	bump singlefile version	2024-08-29 22:33:47 -07:00
Nick Sweeting	57ed2ebcb2	switch youtube_dl install for yt-dlp	2024-08-28 03:46:45 -07:00
Nick Sweeting	038e9f32af	bump deps versions	2024-08-28 03:03:09 -07:00
Nick Sweeting	4ac980a832	hide binary loading debug output for now	2024-08-28 03:02:54 -07:00
Nick Sweeting	1e73a06ba0	change ABIDModel.created to use AutoTimeField seeded on .save instead of auto_now_add so that ts_src for ABID is available on creation before DB row is created	2024-08-28 03:02:37 -07:00
Nick Sweeting	7a734ce410	fix archivebox setup User model import	2024-08-28 03:01:35 -07:00
Nick Sweeting	e558d71b10	fix uuid checks on save	2024-08-28 03:01:19 -07:00
Nick Sweeting	6456cb1727	fix NOT NULL constraint failed: core_snapshot.created_by_id	2024-08-28 00:51:16 -07:00
Nick Sweeting	b789be098d	fix type hint	2024-08-28 00:26:37 -07:00
Nick Sweeting	d0fefc0279	add chunk_size=500 to more iterator calls	2024-08-27 19:28:00 -07:00
Nick Sweeting	44849e1ba2	fix python vendor dependendcies	2024-08-26 20:24:25 -07:00
Nick Sweeting	d7e333b5cf	fix settings.py for prod	2024-08-26 20:20:33 -07:00
Nick Sweeting	24fe958ff3	massively improve Snapshot admin list view query performance	2024-08-26 20:16:43 -07:00
Nick Sweeting	6c4f3fc83a	fix chrome headless=new arg	2024-08-26 20:15:36 -07:00
Nick Sweeting	6ffa710bb3	fix headers Elapsed timedelta is not a JSON-serializable	2024-08-26 20:15:22 -07:00
Nick Sweeting	917af4b1d7	add pydantic logfire and improve settings for debugging	2024-08-26 20:14:47 -07:00
Nick Sweeting	3a57c28ca8	bump pydantic_pkgr version to 0.1.4	2024-08-23 02:03:26 -07:00
Nick Sweeting	5fe3edd79a	new plugin loading system	2024-08-23 02:02:34 -07:00
Nick Sweeting	34389e5e7c	improve CSRF_TRUSTED_ORIGINS loading logic	2024-08-23 02:01:40 -07:00
Nick Sweeting	c6a80ab256	use pydantic_pkgr to parse semver versions	2024-08-23 02:01:24 -07:00
Nick Sweeting	2c2d034d6d	move to new vendoring fallback logic	2024-08-23 02:01:02 -07:00
Nick Sweeting	9c35f3ddb7	add new CSRF_TRUSTED_ORIGINS config option	2024-08-22 18:40:47 -07:00
Nick Sweeting	1a03db2b1d	fix api auth endpoint checking wrong user field	2024-08-22 18:28:23 -07:00
Nick Sweeting	73a3e6aad0	handle tag with no slug or name	2024-08-22 18:25:15 -07:00
Nick Sweeting	cec8015b5b	fix missing STDOUT	2024-08-22 17:58:58 -07:00
Nick Sweeting	38ca5c3228	add extra info to headers.json	2024-08-22 17:57:40 -07:00
Nick Sweeting	80d7a30f13	improve version_str detection	2024-08-22 17:57:33 -07:00
Nick Sweeting	1d31b88fa3	fix migration failing when Tag name is empty	2024-08-22 16:30:25 -07:00
Nick Sweeting	9ecc2ff91c	bump more deps	2024-08-22 15:36:11 -07:00
Nick Sweeting	09553d8340	hardcode EXTRACTOR_CHOICES to prevent nondeterministic migrations	2024-08-22 15:36:02 -07:00
Nick Sweeting	0a5b22700c	bump deps	2024-08-22 15:20:52 -07:00
Nick Sweeting	afe1307617	fix created_by field migration to create User properly if none exists	2024-08-22 15:20:36 -07:00
Nick Sweeting	9b1659c72f	make created_by_id autoapply to any ArchiveResults created under Snapshot	2024-08-20 19:43:07 -07:00
Nick Sweeting	c30ae1d2cb	add created_by_id to all Snapshot creation functions	2024-08-20 19:28:28 -07:00
Nick Sweeting	aa282daadf	show CONSTANTS in separate section of django admin config	2024-08-20 19:27:21 -07:00
Nick Sweeting	ea81f2fc14	use ABIDAdmin as base ModelAdmin	2024-08-20 19:26:40 -07:00
Nick Sweeting	57e48b432c	show redacted API tokens in django admin	2024-08-20 19:25:16 -07:00
Nick Sweeting	ff619fd31f	add api migrations	2024-08-20 18:31:40 -07:00
Nick Sweeting	0285aa52a0	config and attr access improvements	2024-08-20 18:31:21 -07:00
Nick Sweeting	4ae186dfca	fix ABID generation consistency when self._state.adding is True	2024-08-20 05:56:19 -07:00
Nick Sweeting	9d2116ad9a	migration to make fields non editable	2024-08-20 03:46:04 -07:00
Nick Sweeting	9c8ece4d7c	minor css and APIToken Admin UI fixes	2024-08-20 03:45:49 -07:00
Nick Sweeting	75018ed10b	Merge branch 'dev' into feat/search-fields	2024-08-20 03:37:56 -07:00
Nick Sweeting	05880dba8b	bump version to 0.8.2	2024-08-20 03:36:07 -07:00
Nick Sweeting	267964881c	fix tags editor in Snapshot list view	2024-08-20 03:29:57 -07:00
Nick Sweeting	849b4963a1	add migrations	2024-08-20 01:58:44 -07:00
Nick Sweeting	52a813aa80	fix title display in admin UI and abid filter matching in urls	2024-08-20 01:58:36 -07:00
Nick Sweeting	9273db528e	fix abid generation migrations to be historically consistent	2024-08-20 01:58:19 -07:00
Nick Sweeting	506b3d28d4	fix admin UI TagInline and ArchiveResultInline form POST handling	2024-08-20 01:57:07 -07:00
Nick Sweeting	54acfd9f86	improve REST API filter parameters and pagination	2024-08-20 01:56:37 -07:00
Nick Sweeting	850448b42c	add salt args in more places	2024-08-20 01:56:07 -07:00
Nick Sweeting	c4ef2993b2	update REST API and Admin UI to use new id and old_id exclusively	2024-08-19 20:00:04 -07:00
Nick Sweeting	344e902fc6	migrate SnapshotTag to use new snapshot id	2024-08-19 19:42:25 -07:00
Nick Sweeting	74c11d41be	dont allow modifying Snapshot.timestamp as it would break folder dirs	2024-08-19 19:42:01 -07:00
Nick Sweeting	c68a66f74e	add salt kwarg to abid generation funcs	2024-08-19 19:41:43 -07:00
Nick Sweeting	a49739b41c	add hacky workaround for TagInline not showing in admin Snapshot change view	2024-08-19 18:36:35 -07:00
Nick Sweeting	cf2faecf61	add migrations for SnapshotTag through model	2024-08-19 18:36:20 -07:00
Nick Sweeting	7164fb961c	hotlink to snapshots using generic search query link instead of id__startswith	2024-08-19 18:35:35 -07:00
Nick Sweeting	3148d2a3ef	add squashmigrations to allowed mgmgt command list	2024-08-19 18:35:07 -07:00
Nick Sweeting	2c157f0b5b	increase max POST field limit for admin	2024-08-19 18:34:52 -07:00
Nick Sweeting	18ebaed7ed	fix admin UI display of tags	2024-08-19 18:34:01 -07:00
Nick Sweeting	57d31b2b14	fix snapshot uuid	2024-08-18 01:07:21 -07:00
Nick Sweeting	8c50257fe9	move snapshot id to old_id	2024-08-18 00:24:38 -07:00
Nick Sweeting	951025228f	add tag endpoint	2024-08-18 00:24:14 -07:00
Nick Sweeting	a271bcb4ce	use new ids for ArchiveResult API	2024-08-17 23:03:43 -07:00
Nick Sweeting	f72debfdb2	migrate ArchiveResult.id to old_id, and make uuid main id	2024-08-17 22:58:35 -07:00
Nick Sweeting	033ec08d0c	save snapshot ids during migration	2024-08-17 21:56:45 -07:00
Nick Sweeting	4d0bbfccfc	add uri salt and fix api url namespaces	2024-08-17 21:56:23 -07:00
Nick Sweeting	fba3995d86	reorder version output	2024-08-17 19:38:51 -07:00
Nick Sweeting	6a33de6715	remove archive.today link and add JSON	2024-08-17 19:31:43 -07:00
Nick Sweeting	3a87a7fb8c	re-arrange snapshot detail page preference and add opus support	2024-08-17 19:31:34 -07:00
Nick Sweeting	5e0cc926f1	show ulid in archivebox admin	2024-08-17 19:31:18 -07:00
Nick Sweeting	b7745ae37a	add uri salt when creating ABID	2024-08-17 19:30:58 -07:00
Lucas Schwiderski	10f0c4a077	Implement searching individual fields ... So far only for the public view, since we already have a custom search form there, where this is easy to add. This initial implementation supports the common set of metadata fields that the searchbar placeholder also mentions, but adding more fields is trivial.	2024-06-19 16:50:16 +02:00
Nick Sweeting	99e6f0c93f	bump versions	2024-06-04 04:17:33 -07:00
Nick Sweeting	de489d3c60	minor snapshot details ui fixes and migrations log msg improvements	2024-06-04 04:17:32 -07:00
Nick Sweeting	f574d34357	wrap migrations maker in try catch	2024-06-03 03:02:00 -07:00
Nick Sweeting	c63917a22d	fix more CHROME_USER_DATA_DIR issues	2024-06-03 02:51:11 -07:00
Nick Sweeting	1cd62ecc61	add filesizes and stray files in snapshot dir to snapshot_live ui	2024-06-03 02:31:35 -07:00
Nick Sweeting	729f05ab93	fix snapshot_live iframe scrollTo not working due to origin sandbox rules	2024-06-02 19:15:12 -07:00
Nick Sweeting	11a24d9640	merge migrations between branches	2024-06-02 19:14:58 -07:00
Nick Sweeting	b3f2a71c93	bump requirements and version to 0.8.1	2024-06-02 19:14:45 -07:00
Nick Sweeting	07d80db5aa	fix CHROME_USER_DATA_DIR split str error	2024-06-02 19:14:20 -07:00
Nick Sweeting	f72bae8eec	autodiscover plugins on startup	2024-05-17 20:48:11 -07:00
Nick Sweeting	48becde9b4	add new pydantic-based plugin system	2024-05-17 20:13:54 -07:00
Nick Sweeting	e4176dbf7a	change live snapshot preview iframe sandbox rules	2024-05-17 20:12:37 -07:00
Nick Sweeting	774ce3fda7	fix singlefile extractor exception when result is none	2024-05-17 20:12:18 -07:00
Nick Sweeting	29c7aa26bc	show original section titles in config admin ui	2024-05-17 20:11:54 -07:00
Nick Sweeting	acfd346440	make abids searchable in the admin ui	2024-05-17 20:11:31 -07:00
Nick Sweeting	a1afd0211f	fix abid calculation	2024-05-17 20:11:00 -07:00
Nick Sweeting	fdf6f465db	add docstrings	2024-05-14 18:46:38 -07:00
Nick Sweeting	406f57031a	add API support for obj.pk .uuid .abid	2024-05-13 08:52:02 -07:00
Nick Sweeting	a4cc10d7f8	add migrations for third round of field changes	2024-05-13 07:50:22 -07:00
Nick Sweeting	241a7c6ab2	add created, modified, updated, created_by and update django admin	2024-05-13 07:50:07 -07:00
Nick Sweeting	1ba8215072	only use domain part of uri for hash	2024-05-13 07:49:36 -07:00
Nick Sweeting	206e7e74b3	add migrations to create and populate ABIDField and UUIDField values	2024-05-13 05:13:42 -07:00
Nick Sweeting	0420662174	switch everywhere to use Snapshot.pk and ArchiveResult.pk instead of id	2024-05-13 05:12:12 -07:00
Nick Sweeting	9733b8d04c	remove accidentally commited db	2024-05-13 02:38:02 -07:00
Nick Sweeting	4f9f22e024	create abid_utils with new ABID type for ArchiveBox IDs	2024-05-13 02:37:48 -07:00
Nick Sweeting	f896e5dbeb	switch from monkey patching WebhookModel to using swappable	2024-05-13 02:36:15 -07:00
Nick Sweeting	e97d779cd3	move monkey patches to dedicated file	2024-05-13 02:35:19 -07:00
Nick Sweeting	b5ad134264	dont wait for ipython history saver thread before shell exit	2024-05-12 19:25:55 -07:00
Nick Sweeting	ce833e8ead	automatically create storage directories and symlinks based on ulid	2024-05-12 05:21:58 -07:00
Nick Sweeting	33bc4622a0	add ulid and typeid to Snapshot and ArchiveResult	2024-05-12 04:45:34 -07:00
Nick Sweeting	0529099639	create cache dir and sqlite db	2024-05-12 01:42:34 -07:00
Nick Sweeting	e1a04729b3	wait for bg threads to finish before exiting	2024-05-12 01:42:20 -07:00
Nick Sweeting	0438924491	remove unused template vars	2024-05-11 22:43:26 -07:00
Nick Sweeting	2b4b6e5b3a	select single snapshot row automatically when hotlinking to admin actions	2024-05-11 22:33:39 -07:00
Nick Sweeting	27809f2976	fix python3.10 nested fstring quotes	2024-05-11 22:33:27 -07:00
Nick Sweeting	baf24d2d6a	change wording on 404 snapshot detail page	2024-05-11 22:33:17 -07:00
Nick Sweeting	8841e8b181	add new live django template for snapshot detail page	2024-05-11 22:33:02 -07:00
Nick Sweeting	457c42bf84	load EXTRACTORS dynamically using importlib.import_module	2024-05-11 22:28:59 -07:00
Nick Sweeting	c7f55fc3ba	make CHROME_USER_DATA_DIR error quieter	2024-05-11 22:16:10 -07:00
Nick Sweeting	102e87578c	re-lock packages for python3.10 because we have to support it still	2024-05-11 15:06:19 -07:00
Nick Sweeting	913590ee39	explain weird use of ellipses magic value	2024-05-11 15:02:43 -07:00
Nick Sweeting	b90ba6c909	change header auth to use X-ArchiveBox-API-Key so it doesnt conflict with other auth headers	2024-05-08 20:02:47 -07:00
Nick Sweeting	4c5a3fba8b	more fixes for wget_output_path	2024-05-07 05:38:29 -07:00
Nick Sweeting	f2729c9dc7	revert wget character limit back to windows, ascii is allows MORE not less	2024-05-07 05:24:16 -07:00
Nick Sweeting	9b21ce490e	add workaround logic to catch paths that are too long or contain unprintable characters	2024-05-07 05:03:23 -07:00
Nick Sweeting	f62cb5fb43	change wget to use stricter ascii filepath normalization	2024-05-07 05:03:01 -07:00
Nick Sweeting	f770bba3cf	fix OSError 36 caused by checking for path that is too long to exist	2024-05-07 04:12:07 -07:00
Nick Sweeting	ef856e8051	Merge branch 'dev' into issue1316	2024-05-06 23:14:16 -07:00
Nick Sweeting	27d5d1ddc8	revert queryset intersection back to union for search results	2024-05-06 23:13:52 -07:00
Nick Sweeting	f472705d10	Change phrasing (#1419)	2024-05-06 23:11:26 -07:00
Nick Sweeting	3095265880	fix inner quote not escaped	2024-05-06 23:10:59 -07:00
Brandl	8ccd606973	Fix quotation ... Fixes: => ERROR [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt 1.7s ------ > [stage-0 22/23] RUN "/app"/bin/docker_entrypoint.sh version 2>&1 | tee -a /VERSION.txt: 1.665 Traceback (most recent call last): 1.665 File "/usr/local/bin/archivebox", line 5, in <module> 1.665 from archivebox.cli import main 1.665 File "/app/archivebox/cli/__init__.py", line 83, in <module> 1.665 SUBCOMMANDS = list_subcommands() 1.665 ^^^^^^^^^^^^^^^^^^ 1.665 File "/app/archivebox/cli/__init__.py", line 43, in list_subcommands 1.665 module = import_module('.archivebox_{}'.format(subcommand), __package__) 1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1.665 File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module 1.665 return _bootstrap._gcd_import(name[level:], package, level) 1.665 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1.666 File "/app/archivebox/cli/archivebox_add.py", line 11, in <module> 1.666 from ..main import add 1.666 File "/app/archivebox/main.py", line 233 1.666 f'COMMIT_HASH={COMMIT_HASH[:7] if COMMIT_HASH else 'unknown'}', 1.666 ^^^^^^^ 1.666 SyntaxError: f-string: expecting '}'	2024-05-06 21:04:14 +02:00
Nick Sweeting	027c029316	redact passwords, keys, and secret tokens in admin UI	2024-05-06 11:06:42 -07:00
Nick Sweeting	8667ed29f1	improve API webhooks helptext and change app_label to API	2024-05-06 08:11:01 -07:00
Evan Boehs	f998647350	change phrasing	2024-05-06 10:32:36 -04:00
Nick Sweeting	d782bafe2e	fix storages missing stackfiles error	2024-05-06 07:14:01 -07:00
Nick Sweeting	47666ec26b	show webhooks config in django admin	2024-05-06 07:13:54 -07:00
Nick Sweeting	f067451267	fix django timezone.utc removed in 5.0	2024-05-06 07:13:25 -07:00
Nick Sweeting	c7fc9c004f	add django-signal-webhooks	2024-05-06 06:58:03 -07:00
Nick Sweeting	3805a1730d	add 0002 api migration	2024-04-30 21:45:02 -07:00
Nick Sweeting	e20eb52f15	fix COMMIT_HASH missing error	2024-04-30 21:43:22 -07:00
Nick Sweeting	18a5b6e99c	move EXTERNAL_LOCATIONS to data locations in version output	2024-04-25 21:36:43 -07:00
Nick Sweeting	6a6ae7468e	fix lint errors	2024-04-25 21:36:11 -07:00
Nick Sweeting	4a5ad32040	add django-requests-tracker	2024-04-25 18:02:01 -07:00
Nick Sweeting	af669d2f37	rename api files for clarity	2024-04-25 05:55:47 -07:00
Nick Sweeting	75153252dc	big overhaul of REST API, split into auth, core, and cli methods	2024-04-25 03:56:22 -07:00
Nick Sweeting	e5aba0dc2e	fix urljoin bug causing multiple slashes to be merged into one	2024-04-24 19:41:31 -07:00
Nick Sweeting	6cb357e76c	fix fix_url_from_markdown assertion to be valid url	2024-04-24 19:41:11 -07:00
Nick Sweeting	128419f991	expand comment about markdown url trailing paren trimming	2024-04-24 17:50:18 -07:00
Nick Sweeting	beb3932d80	replace uses of URL_REGEX with find_all_urls to handle markdown better	2024-04-24 17:45:45 -07:00
Nick Sweeting	c6d644be29	Merge branch 'dev' into issue1316	2024-04-24 16:24:16 -07:00
Nick Sweeting	98c5e69203	bump lockfiles	2024-04-24 14:38:21 -07:00
Nick Sweeting	8dcfa93ec6	Merge branch 'main' into dev	2024-04-24 14:32:07 -07:00
Nick Sweeting	17f40f3ada	Merge branch 'dev' into fix-URL_REGEX	2024-04-23 19:53:58 -07:00
Nick Sweeting	c6f8a33a63	Update util.py	2024-04-23 19:53:18 -07:00
Nick Sweeting	63fc317229	minor pylint fixes in logging_util	2024-04-23 17:45:36 -07:00
Nick Sweeting	756e159dfe	add new bin/lock_pkgs.sh to generate package lockfiles consistently	2024-04-23 17:45:36 -07:00
longzai	e4dc2701ef	fix URL_REGEX 2	2024-04-11 15:51:55 +08:00
Brandl	5f9aac18f2	api v1	2024-04-10 01:29:24 +02:00
longzai	4ae765ec27	fix the URL_REGEX used in generic_html parsers ... Signed-off-by: longzai <437172242@qq.com>	2024-04-08 04:53:05 +08:00
Nick Sweeting	e48159b8a0	cleanup docker-compose by storing crontabs in data dir	2024-03-26 15:24:05 -07:00
Nick Sweeting	ac73fb5129	merge fixes	2024-03-26 15:22:40 -07:00
Nick Sweeting	b4c3aa5097	Merge branch 'main' into dev	2024-03-26 15:01:36 -07:00
Nick Sweeting	a4453b6f87	fix PERSONAS PERSONAS_DIR typo	2024-03-26 14:19:25 -07:00
jim winstead	8b1b01e508	Update to Django 4.2.x, now in LTS until April 2026	2024-03-25 17:46:01 -07:00
Nick Sweeting	c0b5dbcecb	create new data/personas dir to hold cookies and chrome profiles	2024-03-18 14:41:39 -07:00
Nick Sweeting	c5bb99dce1	explicitly use Default profile inside user data dir	2024-03-18 14:40:40 -07:00
Nick Sweeting	1fc5d7c5c8	add USER_AGENT config option to set all USER_AGENTs at once	2024-03-18 14:39:09 -07:00
jim winstead	5478d13d52	Add generic_jsonl parser ... Resolves #1369	2024-03-14 15:42:29 -07:00
Nick Sweeting	ca2c484a8e	Add _EXTRA_ARGS for various extractors (#1360)	2024-03-14 01:55:09 -07:00
Nick Sweeting	48f4b12ae2	Use COOKIES_FILE to fetch page titles (#1364)	2024-03-14 01:52:44 -07:00
Nick Sweeting	099f7d00fe	Use feedparser for RSS parsing (#1362) ... Fixes #1171 Fixes #870 (probably, would need to test against a Wallabag Atom file to Fixes #135 Fixes #123 Fixes #106	2024-03-14 01:51:45 -07:00
Nick Sweeting	3512dc7e60	Disable searching for existing chrome user profiles by default	2024-03-14 00:58:45 -07:00
Ben Muthalaly	f4deb97f59	Add ARGS and EXTRA_ARGS for Mercury extractor	2024-03-05 21:15:38 -06:00
Ben Muthalaly	d8cf09c21e	Remove unnecessary variable length args for dedupe	2024-03-05 21:13:45 -06:00
Ben Muthalaly	5082d61613	Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file	2024-03-05 02:03:03 -06:00
Ben Muthalaly	4686da91e6	Fix cookies being set incorrectly	2024-03-05 01:48:35 -06:00
Naomi Phillips	a729480b75	Add COOKIES_FILE support for singlefile extractor	2024-03-03 02:32:46 -05:00
Ben Muthalaly	d74ddd42ae	Flip dedupe precedence order	2024-03-01 14:50:32 -06:00
jim winstead	9f462a87a8	Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers ... The feedparser packages has 20 years of history and is very good at parsing RSS and Atom, so use that instead of ad-hoc regex and XML parsing. The medium_rss and shaarli_rss parsers weren't touched because they are probably unnecessary. (The special parse for pinboard is just needing because of how tags work.) Doesn't include tests because I haven't figured out how to run them in the docker development setup. Fixes #1171	2024-03-01 11:25:45 -08:00
Nick Sweeting	a577d1ed23	Merge branch 'dev' into title-cookies-file	2024-02-29 21:29:36 -08:00
Nick Sweeting	fe11e1c2f4	check if COOKIE_FILE is file	2024-02-28 18:19:44 -08:00
jim winstead	ccabda4c7d	Handle list of tags in JSON, and be more clever about comma vs. space	2024-02-28 17:38:49 -08:00
jim winstead	178e676e0f	Fix JSON parser by not always mangling the input ... Rather than by assuming the JSON file we are parsing has junk at the beginning (which maybe only used to happen?), try parsing it as-is first, and then fall back to trying again after skipping the first line Fixes #1347	2024-02-27 14:48:19 -08:00
Ben Muthalaly	68326a60ee	Add cookies file to http request in download_url	2024-02-27 15:30:31 -06:00
Ben Muthalaly	4d9c5a7b4b	Add CHROME_EXTRA_ARGS ... Also fix `YOUTUBEDL_EXTRA_ARGS`.	2024-02-23 18:40:03 -06:00
Ben Muthalaly	ab8f395e0a	Add YOUTUBEDL_EXTRA_ARGS	2024-02-23 15:40:31 -06:00
Ben Muthalaly	4e69d2c9e1	Add EXTRA_*_ARGS for wget, curl, and singlefile	2024-02-22 23:04:11 -06:00
Nick Sweeting	8b9bc3dec8	minor fixes	2024-02-22 04:50:22 -08:00
Nick Sweeting	6a4e568d1b	new archivebox update speed improvements	2024-02-22 04:50:22 -08:00
Nick Sweeting	903c72fa88	fix typing errors	2024-02-12 21:28:52 -08:00
Nick Sweeting	91c4641199	skip dir size calculation when path is too long	2024-02-12 21:26:34 -08:00
Nick Sweeting	19aefc85e6	fix get_system_user failing on uid 999 in k3s	2024-02-08 18:58:12 -08:00
Nick Sweeting	a4bd441077	Update config.py fix trim-filenames	2024-01-31 01:59:43 -08:00
Nick Sweeting	e8772513ca	Update archivebox/core/auth_ldap.py ... add log line	2024-01-24 11:00:42 -08:00
Vladimir D	914df75ee7	Merge branch 'dev' into feature/is_staff-is_superuser-flags-defaults	2024-01-24 22:25:27 +04:00
Vladimir D	95580ee743	populate is_staff and is_superuser flags at LDAP authentication	2024-01-24 22:18:02 +04:00
Nick Sweeting	3b36928bdc	add yt-dlp restrict-filenames option to make filenames shorter	2024-01-23 14:20:59 -08:00
Nick Sweeting	93781c58ce	add more git clone domains	2024-01-23 13:51:56 -08:00
Nick Sweeting	c7cab7cadc	Show the upgrade notification only in specific views (#1314)	2024-01-23 10:29:14 -08:00
Neel Suthar	279883d6bb	Text Search and Filters don't work at the same time in the web UI #1316 ... Making sure to return distinct results. Changing set operation to '&' to show the matching results from filters AND search term	2024-01-21 17:34:22 -06:00
Neel Suthar	aca0512926	Fix for Bug: Search sometimes shows the same snapshot twice #1193 ... Making sure the search results are unique	2024-01-19 17:03:51 -06:00
Neel Suthar	702b81f9e6	Fixes #1193 Bug: Search sometimes shows the same snapshot twice ... Making sure user gets the distinct set of search results	2024-01-19 13:49:09 -06:00
Nick Sweeting	d0cd84a2af	Merge branch 'dev' into specific-version-banner	2024-01-19 04:01:32 -08:00
Nick Sweeting	1773146833	include more output file locations when considering whether snapshot.is_archived	2024-01-19 03:47:38 -08:00
Nick Sweeting	0a25495520	add fallback to check wget output dir with port stripped	2024-01-19 03:47:38 -08:00
Nick Sweeting	ef1a697588	fix sorting by Size or by Files to sort by number of archive results	2024-01-19 03:47:38 -08:00
Nick Sweeting	2e77c3964f	persist snapshot index header collapse state	2024-01-19 03:47:38 -08:00
Nick Sweeting	f839f4f16e	Update navigation.html to fix target blank inside django tag typo	2024-01-16 00:37:55 -08:00
Nick Sweeting	8e03755a7a	Make "DOCS" link in navigation bar open in new tab (#1323)	2024-01-15 12:36:51 -08:00
Tristan Sherliker	d6a8e12482	Update navigation.html ... change navigation template so docs link opens in new tab	2024-01-14 16:40:11 +00:00
Nick Sweeting	8c07b7e127	disable automatic chrome selfupdating	2024-01-11 19:51:27 -08:00
Nick Sweeting	9766de1b70	bump version of dev build to 0.7.3	2024-01-11 19:08:13 -08:00
Nick Sweeting	6184f659dc	improve window size chrome cli handling	2024-01-11 19:02:46 -08:00
Ben Muthalaly	5de45dbf30	Show upgrade notification in admin snapshot view	2024-01-08 22:55:30 -06:00
Nick Sweeting	1532186283	Merge branch 'main' into dev	2024-01-04 12:30:46 -08:00
Nick Sweeting	a215b38681	update dompurify version	2024-01-04 11:43:50 -08:00
Nick Sweeting	94b132818d	fix lint errors	2024-01-04 11:41:12 -08:00
Nick Sweeting	c1fd2cfa42	tag URLs immediately once added instead of waiting until archival completes	2024-01-03 20:31:46 -08:00