823 commits

Author	SHA1	Message	Date
Nick Sweeting	104553489f	remove redundant utils file	2020-11-28 02:12:27 -05:00
Nick Sweeting	84507b68b5	add legacy code warning to schema.py	2020-11-28 02:03:40 -05:00
Nick Sweeting	7fdea91311	fix static html num_outputs info	2020-11-28 02:01:53 -05:00
Nick Sweeting	fde65c3b7d	fix public index missing template context	2020-11-28 01:29:34 -05:00
Nick Sweeting	7d7ce3a790	fix Snapshot count in column header	2020-11-28 01:22:58 -05:00
Nick Sweeting	9fc965d3da	remove broken json download link	2020-11-28 01:19:01 -05:00
Nick Sweeting	46a53eafdb	simplify history helper	2020-11-28 01:14:45 -05:00
Nick Sweeting	c9162a6d09	remove finished/not finished spinners	2020-11-28 01:07:02 -05:00
Nick Sweeting	9661c863b3	css style tweaks for icons	2020-11-28 01:06:23 -05:00
Nick Sweeting	910f3d65c7	default function args can never be mutable	2020-11-28 01:06:11 -05:00
Nick Sweeting	411fdcac87	use database for num_outputs instead of legacy json	2020-11-28 01:05:53 -05:00
Nick Sweeting	7f2c834ea3	fix check_data_folder mypy types	2020-11-28 01:05:35 -05:00
Nick Sweeting	1b22f8eeef	Merge pull request #515 from cdvv7788/POC-setup-django-on-init	2020-11-27 23:56:37 -05:00
Nick Sweeting	00bb55203e	always show WARC icon with opacity set based on exists	2020-11-27 23:45:49 -05:00
Nick Sweeting	efe3027797	Merge branch 'master' into archive-result	2020-11-27 23:18:11 -05:00
Nick Sweeting	e4d2ac432d	improve OS kernel output in archivebox version	2020-11-27 23:08:23 -05:00
Nick Sweeting	07a56f9d46	also print platform and CPU info in version output	2020-11-27 22:59:18 -05:00
Nick Sweeting	d9ef3d0bf8	ignore lost+found dir in data folder	2020-11-27 19:39:19 -05:00
Cristian	4b3f72202b	feat: Bump django, update migration and change cmd to use JSONField	2020-11-27 16:23:27 -05:00
Cristian	f61e6a74bb	feat: Re-add unused icons in list view	2020-11-27 15:55:37 -05:00
Nick Sweeting	f84f288bef	Apply suggestions from code review ... minor nit	2020-11-27 00:01:34 -05:00
Nick Sweeting	5e7c2d0ab8	show archivebox and node versions in version cmd output	2020-11-23 20:24:44 -05:00
mAAdhaTTah	be7a7f8548	Fix string checks in schedule ... `s` comes through as a `PosixPath`, so both the `' ' in s` & return value, later used by `join`, complain.	2020-11-23 18:34:07 -05:00
Cristian	34a1a6d30d	fix: Update model according to code review	2020-11-23 18:28:43 -05:00
Nick Sweeting	02551c0152	minor packaging fixes and bump to 0.4.21	2020-11-23 17:28:45 -05:00
Nick Sweeting	83693a5c03	add packaging setup with stdeb for debian and apt ... vendor the base32_crockford lib add build script for debain packages	2020-11-23 16:57:05 -05:00
Nick Sweeting	0e2ccbc10d	update urls to new repo path	2020-11-23 02:06:46 -05:00
Nick Sweeting	b11d562445	fix splitting on multiple equals in val	2020-11-22 12:33:15 -05:00
Nick Sweeting	afe9319c25	Merge pull request #537 from TrAyZeN/master	2020-11-18 23:20:41 -05:00
Nick Sweeting	d32b27abcb	Merge pull request #540 from jdcaballerov/hotfix-search-fields ... hotfix: Fixes 500 error on Admin search	2020-11-17 10:36:18 -05:00
JDC	8b0250caeb	Fixes 500 error on search ... The class SnapshotAdmin search_fields includes the tags ManyToMany field causing a django.core.exceptions.FieldError: Related Field got invalid lookup: icontains error. A related search field tags__name should be used.	2020-11-17 08:36:03 -05:00
TrAyZeN	88cc75a045	Change opacity of inexisting archive type on public view	2020-11-14 17:48:29 +01:00
TrAyZeN	a05485f85c	Fix file icons order	2020-11-14 17:44:06 +01:00
Nick Sweeting	fdd4effc92	Merge pull request #535 from cdvv7788/extractors-flag	2020-11-13 14:53:17 -05:00
Nick Sweeting	257d3f2a98	Update archivebox/cli/archivebox_add.py	2020-11-13 14:52:21 -05:00
JDC	d54c3eec9d	Add tag filter argument to remove command	2020-11-13 14:16:48 -05:00
Cristian	54df0a035b	fix: Move csv split to the add function to avoid optional nullable argument	2020-11-13 13:10:17 -05:00
Cristian	1ec8276514	fix: Use a comma separated input instead of nargs for the extract flag	2020-11-13 13:01:11 -05:00
JDC	cbb3d04c12	Allow list filtering by tag name	2020-11-13 12:06:12 -05:00
Cristian	db523c9d82	fix: Avoid mutable default input argument	2020-11-13 11:41:50 -05:00
Cristian	44eede96e5	feat: Add extract flag to add command	2020-11-13 09:24:34 -05:00
Nick Sweeting	4372cb6eec	stop execution entirely when atomic_write is unsupported	2020-11-12 14:55:21 -05:00
Nick Sweeting	3f160eab8e	correctly handle WGET_AUTO_COMPRESSION failing when wget is missing	2020-11-12 14:28:43 -05:00
Cristian	0f13087a09	refactor: Remove unneeded prefetch related	2020-11-12 13:58:13 -05:00
Cristian	c565fad75c	feat: Use prefetch related to reduce the number of queries to the database on public index view	2020-11-12 11:37:56 -05:00
Cristian	8cfad64271	feat: Add specific logic for archive_org icon	2020-11-12 11:09:34 -05:00
Cristian	e594e6a75a	feat: WARC link points to the first warc result in target path	2020-11-12 10:57:31 -05:00
Cristian	b237e412df	feat: Finish reversal. Add ArchiveResults that are not found in the index.json	2020-11-12 10:30:41 -05:00
Cristian	f7f0bebdcc	feat: Modify migration reverse function to restore index (WIP)	2020-11-11 15:26:54 -05:00
Cristian	508a0bb06e	refactor: Unpack extractors tuple instead of using the index to access the relevant information	2020-11-10 12:38:29 -05:00
Nick Sweeting	fbd9a7caa6	add explicit error when FSYNC is not supported on filesystem	2020-11-10 01:07:56 -05:00
Cristian	71655220ad	feat: Add warc to list and limit check to succeeded archive results	2020-11-05 07:54:40 -05:00
Cristian	33182fd53c	fix: Add missing assignation	2020-11-04 15:07:45 -05:00
Cristian	d064a3eeff	fix: Handle case when update tries to re-add a link that is not in the sql index	2020-11-04 15:02:54 -05:00
Cristian	f292cface2	fix: Add condition for oneshot when archiving links	2020-11-04 14:40:44 -05:00
Cristian	4484491fb7	feat: Create ArchiveResult after finishing an extractor process	2020-11-04 11:22:55 -05:00
Cristian	b3e0400bc0	feat: initial functional version with icons calculated based on archive results	2020-11-04 10:31:20 -05:00
Cristian	309a87e8fe	feat: Add extractor field to the database	2020-11-04 07:28:02 -05:00
Cristian	8f3c03a0f9	feat: Initial (and naive) ArchiveResult model	2020-11-03 09:54:02 -05:00
Cristian	ac0ec160d1	lint: Fix warnings in master branch	2020-11-02 08:51:48 -05:00
Nick Sweeting	7d4738a674	fix intermittent BrokenPipe error on macOS when SHOW_PROGRESS=True	2020-10-31 19:38:54 -04:00
Nick Sweeting	9c6ff5036c	add suppress output helper	2020-10-31 19:33:17 -04:00
Nick Sweeting	22fb9c2ad7	tweak icons	2020-10-31 19:32:43 -04:00
Nick Sweeting	cafe35c595	show pending in light font	2020-10-31 16:33:31 -04:00
Nick Sweeting	5cae05ae76	tweak tags css and add tags to navbar	2020-10-31 07:57:11 -04:00
Nick Sweeting	c47398851b	nicer timeout hints	2020-10-31 07:57:11 -04:00
Nick Sweeting	651d6c4447	bold snapshots over 50MB	2020-10-31 07:57:11 -04:00
Nick Sweeting	b8bbb75f9c	logarithmic progress bars woohoo	2020-10-31 07:57:11 -04:00
Nick Sweeting	ac9e0e356d	config fixes	2020-10-31 07:57:11 -04:00
Nick Sweeting	79051ca15b	new package build	2020-10-31 03:08:41 -04:00
Nick Sweeting	18355dc2c6	clean up config loading in settings and config file layout	2020-10-31 03:08:03 -04:00
Cristian	e7e33ea7a5	tests: Add tests for several different ways to extract the title	2020-10-30 08:04:26 -05:00
Nick Sweeting	aede134ab3	temporarily disable icon highlighting in favor of performance	2020-10-30 05:12:33 -04:00
Nick Sweeting	f727ece7b3	add regex fallback back to title parser	2020-10-30 04:57:31 -04:00
Nick Sweeting	79bef1384e	Merge pull request #493 from ttimasdf/feat-ogtitle ... Feature: add og:title metadata as alternative title	2020-10-30 04:51:14 -04:00
Nick Sweeting	cac3912439	small type fixes	2020-10-30 04:50:14 -04:00
Nick Sweeting	1e5fbf4bd2	Update archivebox/config/__init__.py	2020-10-29 13:46:03 -04:00
Cristian	81dd626b85	fix: CHROME_USER_DATA_DIR was causing an error after the update to posix paths	2020-10-29 11:09:18 -05:00
Cristian	a6bee5f111	feat: Move setup_django to an inner module	2020-10-26 08:02:04 -05:00
Cristian	e1d0b8bce7	feat: Initialize django at the beginning	2020-10-26 07:45:21 -05:00
Nick Sweeting	5faadee7d1	workaround for mercury version output	2020-10-24 22:59:09 -04:00
Nick Sweeting	e727af6f22	allow Path args to get_dir_size and copy_and_overwrite	2020-10-24 22:47:18 -04:00
Cristian	f330e6428b	lint: Remove unused imports from utils	2020-10-23 06:45:56 -05:00
Cristian	f397634dd2	feat: Rename old indexes at the end of init process	2020-10-23 06:45:56 -05:00
Cristian	7fc9b7d456	refactor: Update mentions of the html index in the logs	2020-10-23 06:45:56 -05:00
Cristian	572b46cecf	lint: Remove unused imports	2020-10-23 06:45:56 -05:00
Cristian	ae1484b8bf	feat: Remove index.json and index.html generation from the regular process	2020-10-23 06:45:56 -05:00
Nick Sweeting	494af5f2e1	Merge pull request #507 from ehainry/master ... Add parser for Wallabag Atom feeds	2020-10-22 14:04:57 -04:00
Cristian	14f56a868a	refactor: Change typing for new stubs	2020-10-22 08:46:16 -05:00
Cristian	c12fe0e3d7	feat: Use CURL_ARGS on title extractor	2020-10-22 08:46:16 -05:00
Cristian	563d0f94ec	feat: Use CURL_ARGS in favicon extractor	2020-10-22 08:46:16 -05:00
Cristian	2e1cdca789	feat: Use CURL_ARGS on header extractor	2020-10-22 08:46:16 -05:00
Cristian	972d57bd08	feat: Add CURL_ARGS to control curl arguments	2020-10-22 08:46:16 -05:00
Cristian	24e7a74855	feat: Add WGET_ARGS to control wget arguments	2020-10-22 08:46:16 -05:00
Cristian	65530e1e5b	refactor: Use json.loads instead of split for list arguments	2020-10-22 08:46:16 -05:00
Cristian	bc02e0ffe3	feat: Add config for youtubedl (YOUTUBEDL_ARGS)	2020-10-22 08:46:16 -05:00
Cristian Vargas	a850b4a9d9	Merge branch 'master' into tags	2020-10-20 08:23:25 -05:00
Emmanuel Hainry	aebc83659d	Add parser for Wallabag Atom feeds	2020-10-18 11:20:07 +02:00
Cristian	62c78e1d10	refactor: Remove django-taggit and replace it with a local tags setup	2020-10-12 13:47:03 -05:00
Nick Sweeting	6c704fa8cf	Merge pull request #498 from adamwolf/bookmarklet ... Add a bookmarklet	2020-10-09 21:59:07 -04:00