363 commits

Author	SHA1	Message	Date
Nick Sweeting	834aaa1591	better template staticfile management with themes dir	2019-04-22 19:07:39 -04:00
Nick Sweeting	f0f516e853	check for data folder when running most subcommands	2019-04-22 19:06:48 -04:00
Nick Sweeting	2f0dbeebc1	update docstrings and comments	2019-04-22 14:42:04 -04:00
Nick Sweeting	ab68819332	add archivebox info command to scan data dir	2019-04-22 14:34:30 -04:00
Nick Sweeting	50b947f41d	add md5 hashes to dependencies dict	2019-04-22 14:34:12 -04:00
Nick Sweeting	29ced7b5c8	allow running archivebox core commands from manage.py	2019-04-22 13:36:27 -04:00
Nick Sweeting	168e578ea4	fix bad default in scheduler	2019-04-22 13:21:08 -04:00
Nick Sweeting	354895aef1	django admin to view links now working	2019-04-22 13:20:19 -04:00
Nick Sweeting	f489dd96a9	fix archivebox remove rejecting stdin patterns	2019-04-22 13:19:47 -04:00
Drewry Pope	332a32f4f9	Resolve 3 typos in util.py	2019-04-20 02:59:44 -05:00
Nick Sweeting	39a0ab3013	add pipenv, schedule cmd, logs dir, and lots more	2019-04-18 21:10:00 -04:00
Nick Sweeting	4f869f235f	add package headers	2019-04-17 22:00:54 -04:00
Nick Sweeting	920898e160	working archivebox-server command	2019-04-17 05:42:35 -04:00
Nick Sweeting	669bd6bee4	first views for archivebox server	2019-04-17 05:42:21 -04:00
Nick Sweeting	289a6ea30f	fix database file location and init process	2019-04-17 05:42:09 -04:00
Nick Sweeting	88a37bc552	fix json list output	2019-04-17 05:41:41 -04:00
Nick Sweeting	35aa8c8902	clearer sql parsing and dumping	2019-04-17 03:50:41 -04:00
Nick Sweeting	cdb70c73df	first working django model with archivebox-shell command and sql exporting	2019-04-17 03:49:18 -04:00
Nick Sweeting	ecf95d398a	cleanup after test output	2019-04-17 02:27:38 -04:00
Nick Sweeting	8101ce7f23	add tests	2019-04-17 02:25:39 -04:00
Nick Sweeting	9ce47431da	better loading and saving storage mechanism	2019-04-17 02:25:28 -04:00
Nick Sweeting	c95f893b61	use out_dir instead of link_dir	2019-04-17 02:23:45 -04:00
Nick Sweeting	8b4b13b667	accept multiple pattern args for removal	2019-04-17 02:22:29 -04:00
Nick Sweeting	21174da014	better data folder checking on startup	2019-04-17 02:22:19 -04:00
Nick Sweeting	22bea7a4f6	use atomic writes inside to_json helper func	2019-04-16 23:22:24 -04:00
Nick Sweeting	717e390ef6	remove derived link info in favor of derived Link properties	2019-04-16 23:21:24 -04:00
Nick Sweeting	50d368b1bc	log matching links in a more logical place	2019-04-16 23:20:31 -04:00
Nick Sweeting	1e759084f3	dedupe urls using exact url instead of fuzzy url	2019-04-16 23:19:44 -04:00
Nick Sweeting	6e5a77e1ad	check data folder on startup	2019-04-16 23:19:21 -04:00
Nick Sweeting	fafdef1e6d	prevent running as root	2019-04-16 23:18:42 -04:00
Nick Sweeting	3fb10dbf35	working better removal ux	2019-04-11 08:11:32 -04:00
Nick Sweeting	525f8beb55	better no links found message	2019-04-11 07:00:34 -04:00
Nick Sweeting	d8d8f7c2cc	working consistent list and remove with filtering	2019-04-11 07:00:26 -04:00
Nick Sweeting	4ca9a0beac	colorized and better command sorting in help msg	2019-04-11 06:59:30 -04:00
Nick Sweeting	718e25c973	better dependency checking system and guards	2019-04-11 03:42:35 -04:00
Nick Sweeting	fafe6e75c5	fix version in footer linking to git sha instead of release tag	2019-04-11 03:42:12 -04:00
Nick Sweeting	d08978d66c	always hide progress bar even when exceptions are thrown	2019-04-11 03:41:50 -04:00
Nick Sweeting	0272c9b8c0	deduplicate method history when merging links	2019-04-11 03:41:25 -04:00
Nick Sweeting	0d2f7eb58e	expand user tildes in paths	2019-04-11 03:41:05 -04:00
Nick Sweeting	bcfe17bc87	define database file in config.py	2019-04-11 03:40:37 -04:00
Nick Sweeting	b69f26297b	restrict wget filenames to windows-fs compatible characters	2019-04-11 03:40:16 -04:00
Nick Sweeting	4f599c0b0b	escape all non-windows-friendly filenames	2019-04-10 22:46:20 -04:00
Nick Sweeting	eb2b6978c3	comment out uninmplemented args	2019-04-03 03:52:56 -04:00
Nick Sweeting	fd80275856	print import instructions on first run	2019-04-03 01:54:15 -04:00
Nick Sweeting	749f06fe5b	simplify bin shortcut	2019-04-03 00:29:58 -04:00
Nick Sweeting	51ae634ec9	working argparse based CLI with most commands implemented	2019-04-03 00:27:37 -04:00
Nick Sweeting	68b4c01c6b	working archivebox command inside django legacy folder	2019-04-02 18:53:21 -04:00
Nick Sweeting	27708152d2	wip initial django setup	2019-04-02 16:36:41 -04:00
luoliyan	924de7f68c	Update purge script to match codebase cleanup	2019-04-02 13:13:07 +09:30
Nick Sweeting	03f300d050	always patch all links	2019-03-30 23:47:56 -04:00
Nick Sweeting	f1075f2c7d	fix links index	2019-03-30 23:43:53 -04:00
Nick Sweeting	97249a1861	handle urls with special characters properly	2019-03-30 22:25:10 -04:00
Nick Sweeting	6a8f6f52af	0 mypy errors	2019-03-30 21:29:16 -04:00
Nick Sweeting	f4e018ba0c	fix a bunch of mypy errors	2019-03-30 20:49:45 -04:00
Nick Sweeting	eef8ca29f0	hide compression detection failure during config setup	2019-03-30 20:31:28 -04:00
Nick Sweeting	ac9fed06fd	whitespace	2019-03-30 17:59:56 -04:00
Nick Sweeting	880b425df6	add note about saving timestamp strings independently	2019-03-30 17:57:39 -04:00
Nick Sweeting	6d92784dbb	add docstring about timestamp parsing	2019-03-30 17:51:23 -04:00
Nick Sweeting	8283b353f4	fix wget_auto_compression check prining stderr to console	2019-03-30 17:43:44 -04:00
Nick Sweeting	fffeb21ad4	remove progress dots in title fetching	2019-03-30 15:50:32 -04:00
Nick Sweeting	1e45b02684	add flake8 config	2019-03-30 15:41:26 -04:00
Nick Sweeting	03a388300f	fix link parsing	2019-03-30 15:38:28 -04:00
Nick Sweeting	b58afcf974	fix broken import	2019-03-30 15:37:36 -04:00
Nick Sweeting	5d0185b6dd	Merge branch 'master' into dev	2019-03-30 15:36:54 -04:00
Nick Sweeting	1191cf1df1	fix importerror of scheme	2019-03-30 15:04:43 -04:00
Nick Sweeting	73f46b0b29	add proper typechecked json parsing and dumping	2019-03-30 15:03:46 -04:00
Nick Sweeting	35c05c321f	minor bin version checking changes	2019-03-30 15:03:31 -04:00
Nick Sweeting	529a0f8bb2	fix broken function name	2019-03-30 15:00:21 -04:00
Nick Sweeting	066b36b6a9	make URL_BLACKLIST empty by default	2019-03-30 14:56:19 -04:00
Nick Sweeting	9cdceecda8	final link rewrite on ctrl+c	2019-03-27 20:49:09 -04:00
Nick Sweeting	4c8e45b8d7	save all imports to sources dir	2019-03-27 20:48:41 -04:00
Nick Sweeting	cc3d1e9cc9	limit length of stringified arg_vals in exceptions	2019-03-27 18:26:22 -04:00
Nick Sweeting	8b50fee0f5	better type checking of latest output methods	2019-03-27 18:25:17 -04:00
Nick Sweeting	d2a34f2602	switch to atomic disk writes for all index operations	2019-03-27 18:24:57 -04:00
Nick Sweeting	a214bd7c02	make everything take link_dir as an optional arg since its derivable from link url	2019-03-27 18:24:30 -04:00
Nick Sweeting	9fc1e3c3e1	provide migration hint when initializing	2019-03-27 18:21:53 -04:00
Nick Sweeting	a26c2fe467	show full version info using flag	2019-03-27 16:44:00 -04:00
Nick Sweeting	3375522ff4	remove uneeded bin shortcuts	2019-03-27 15:49:41 -04:00
Nick Sweeting	64f92af60b	working setuptools package distribution	2019-03-27 15:39:30 -04:00
Nick Sweeting	93216a3c3e	new version handling and absolute imports	2019-03-27 15:35:13 -04:00
Nick Sweeting	bc1bc9fe02	htmldecode all urls and titles during parsing	2019-03-27 15:33:59 -04:00
Nick Sweeting	b1b0c8d1c5	show prettier failure output during link archiving	2019-03-27 15:33:12 -04:00
Nick Sweeting	ea695b8bef	remove dataclass for ArchiveIndex in favor of plain dict to simplify schema file	2019-03-27 15:32:39 -04:00
Nick Sweeting	88721512d4	more detailed parsing and indexing cli output	2019-03-27 15:16:53 -04:00
Nick Sweeting	b9839500b2	make archivebox use current directory as OUTPUT_DIR by default	2019-03-27 15:15:51 -04:00
Nick Sweeting	0ad8d51f0e	remove base32 crockford in favor of dependency	2019-03-27 15:14:47 -04:00
Nick Sweeting	2e8008009e	make main compatible with setuptools entrypoint api	2019-03-27 15:14:22 -04:00
Nick Sweeting	79b319a48c	make archive.py runnable as package and as module	2019-03-27 15:13:47 -04:00
mlazana	8502fa5cc3	config.py: update function exclude_blacklisted(links)	2019-03-27 20:10:05 +02:00
mlazana	a3705e31c6	Merge remote-tracking branch 'upstream/master'	2019-03-27 20:03:07 +02:00
Nick Sweeting	002206d4fe	remove distutils in favor of shutil	2019-03-27 11:39:51 -04:00
Nick Sweeting	e1a1ea25dd	add readability script	2019-03-27 11:39:28 -04:00
Nick Sweeting	7ae680a461	Merge branch 'master' into dev	2019-03-27 10:36:29 -04:00
Nick Sweeting	0835baf435	show link titles in black	2019-03-27 10:31:44 -04:00
Nick Sweeting	407484b91e	show progress during main index writing	2019-03-27 10:31:22 -04:00
Nick Sweeting	19e1c35f1a	support loading link indexes with extra keys	2019-03-27 10:31:07 -04:00
Nick Sweeting	6bec1709f6	Update config.py	2019-03-27 04:48:38 -04:00
Nick Sweeting	98870ba428	fit bigger titles with tighter spacing	2019-03-27 03:56:38 -04:00
Nick Sweeting	d2cf260e7c	css tweaks	2019-03-27 03:52:38 -04:00
Nick Sweeting	2ed91fe429	new details page design	2019-03-27 03:49:39 -04:00
Nick Sweeting	c9c5b04df0	full type-hinting coverage	2019-03-26 23:25:07 -04:00
Nick Sweeting	ab09560f14	working runtime type casting and enforcement for a wide range of types	2019-03-26 22:26:21 -04:00
Starbeamrainbowlabs	ab72a2dad1	Try to add --compression back again	2019-03-26 23:38:50 +00:00
Starbeamrainbowlabs	51859731d3	config.py: add WGET_AUTO_COMPRESSION	2019-03-26 23:34:45 +00:00
Nick Sweeting	0d8a076c1f	add base32 crockford dependency	2019-03-26 19:21:48 -04:00
Nick Sweeting	25a107df43	switch to dataclasses, working Link type hints everywhere	2019-03-26 19:21:34 -04:00
Starbeamrainbowlabs	ed7ad04fd9	wget: Remove unsupported compression option ... --compression=auto is not supported on Raspberry Pis. This PR removes it from ArchiveBox.	2019-03-26 15:46:57 +00:00
Nick Sweeting	76abc58135	switch to strict type hints with NamedTuples instead of dicts	2019-03-26 05:33:34 -04:00
Nick Sweeting	0a44779b21	save command versions in archive results	2019-03-26 05:32:16 -04:00
Nick Sweeting	52871f42e7	cleaner config with type hints	2019-03-26 05:31:27 -04:00
Nick Sweeting	4f8c99011a	fix terminal resizing making progress bar go crazy	2019-03-26 05:30:23 -04:00
Nick Sweeting	63abc8980c	add mypy type hints	2019-03-26 03:20:41 -04:00
Nick Sweeting	58c9b47d43	fix rss parsing when items have newlines between them	2019-03-25 16:27:55 -04:00
mlazana	81d846427e	fix comments in links.py	2019-03-24 19:04:22 +02:00
mlazana	4d10568477	exclude links that are in blacklist	2019-03-24 14:40:26 +02:00
mlazana	417ee9e302	add env variable URL_BLACKLIST	2019-03-23 21:27:41 +02:00
Nick Sweeting	fb8d4e9369	fix title None error	2019-03-22 23:10:11 -04:00
Nick Sweeting	c28c0d0e71	enable gzip auto decompression	2019-03-22 23:01:01 -04:00
Nick Sweeting	f7a0568a6c	better chrome options loading	2019-03-22 23:00:53 -04:00
Nick Sweeting	ef865dd76a	better stats collection and printing	2019-03-22 23:00:43 -04:00
Nick Sweeting	4c499d77b6	move dependency checking into config file	2019-03-22 22:05:45 -04:00
Nick Sweeting	096832210c	better logging totals counts	2019-03-22 21:38:24 -04:00
Nick Sweeting	d06775923b	move latest to derived data using history	2019-03-22 21:38:08 -04:00
Nick Sweeting	69f837bbf6	simplify chrome_user_data_dir default	2019-03-22 21:37:02 -04:00
Nick Sweeting	8f73fdbe09	fix chrome profile precedence order to be equal	2019-03-22 21:31:55 -04:00
Nick Sweeting	bd9f3e313f	better logging during long output	2019-03-22 15:09:39 -04:00
Nick Sweeting	1c5732d5c6	improve logging of skipped links	2019-03-22 14:01:29 -04:00
Nick Sweeting	0075daa45c	Merge pull request #187 from simon987/master ... Fix succeded typo	2019-03-22 12:35:15 -04:00
simon987	1c7dafb378	Fix for #188	2019-03-21 20:12:27 -04:00
simon987	fa425b563e	Fix succeded typo	2019-03-21 19:59:09 -04:00
Nick Sweeting	d798117081	better separation of archive method checking and running logic	2019-03-21 05:35:41 -04:00
Nick Sweeting	e6bd1f8ca8	major codebase-wide code cleanups	2019-03-21 01:28:12 -04:00
Nick Sweeting	c806068683	fix missing import	2019-03-20 21:12:43 -04:00
Nick Sweeting	5ee1c39720	remove flawed link_type concept in favor of simpler staticfile detection	2019-03-20 21:11:29 -04:00
Nick Sweeting	c79e1df8b2	use hints system to pass stdout and errors up to the traceback	2019-03-20 21:10:09 -04:00
Nick Sweeting	c38de3bbe7	nicer iframes on detail pages	2019-03-20 04:30:44 -04:00
Nick Sweeting	e4c7683983	move js to head and optimize for loading speed	2019-03-20 04:30:28 -04:00
Nick Sweeting	c2ea90ed39	better favicon loading	2019-03-20 04:30:00 -04:00
Nick Sweeting	1772bbfb13	minor ui tweaks	2019-03-20 02:57:06 -04:00
Nick Sweeting	ac56023bfd	fix wget_output_path relative location building	2019-03-20 02:37:27 -04:00
Nick Sweeting	9c166d5bcf	add more files links to index details page	2019-03-20 02:37:13 -04:00
Nick Sweeting	7f84c4b3c8	dont show in progresss message or regen index on exit	2019-03-20 02:36:55 -04:00
Nick Sweeting	f9a7c530b9	fix pinboard RSS parser	2019-03-19 18:44:22 -04:00
Nick Sweeting	61ec4971e9	fix nested archive index page and improve wget output finding	2019-03-19 18:10:11 -04:00
Nick Sweeting	eb5cc8078a	better UX before titles have been fetched during archiving progress	2019-03-19 18:10:11 -04:00
Nick Sweeting	914750c453	better title regex to match titles surrounded by newlines	2019-03-19 18:10:11 -04:00
Nick Sweeting	1b5201fd58	re-save index on Ctrl+c to hide in progress message on html output	2019-03-19 18:10:11 -04:00
noncetonic	28758cf16c	Adds CHROME_USER_AGENT	2019-03-19 10:15:52 -07:00
noncetonic	e230e27929	Changes HEADLESS_USER_AGENT to CHROME_USER_AGENT	2019-03-19 08:13:27 -07:00
noncetonic	a13f22d15a	Adds support for HEADLESS_USER_AGENT for Chrome	2019-03-19 05:32:48 -07:00