633 commits

This branch

This branch

All branches

Author	SHA1	Message	Date
Nick Sweeting	071b39b944	nicer error formats	2019-01-20 14:08:33 -05:00
Nick Sweeting	de6eb649e0	disable wget timestamping if using warcs	2019-01-20 14:08:33 -05:00
Nick Sweeting	ce07a6c738	patch subprocess.run to have better timeout handling	2019-01-20 14:08:33 -05:00
Nick Sweeting	e620722bb9	fix timeouts hanging the progressbar	2019-01-20 14:08:33 -05:00
Nick Sweeting	7ea36c4adb	bump timeouts and improve curl archive method	2019-01-20 14:08:33 -05:00
Nick Sweeting	370adb56bf	Update README.md	2019-01-18 17:45:48 -05:00
Nick Sweeting	ea905d68bb	fix etc nginx and Docker config	2019-01-16 09:20:15 -05:00
Nick Sweeting	d24724ed79	Update Dockerfile	2019-01-16 08:05:01 -05:00
Nick Sweeting	0cb915cd24	Update README.md	2019-01-15 18:35:11 -05:00
Nick Sweeting	cc8611de83	Update config.py	2019-01-14 22:46:35 -05:00
Nick Sweeting	c42fcd42d7	fetch warc file inline with wget instead of as separate step	2019-01-14 22:43:14 -05:00
Nick Sweeting	2d86d4dfe0	update docs	2019-01-14 22:43:14 -05:00
Nick Sweeting	3186f24867	Update README.md	2019-01-14 19:12:08 -05:00
Nick Sweeting	45c499dc9e	Update README.md	2019-01-14 18:19:30 -05:00
Nick Sweeting	300b5c6182	put ArchiveBox and wget version in user agent	2019-01-14 18:17:30 -05:00
Nick Sweeting	39f5e12364	allow passing import list via stdin	2019-01-14 18:11:48 -05:00
Nick Sweeting	cb60bad1d7	disable WARC by default	2019-01-12 20:19:17 -05:00
Nick Sweeting	321cdeda11	add mirror to git clone	2019-01-12 20:13:13 -05:00
Nick Sweeting	c5dcb68a6d	install youtubedl in setup	2019-01-12 03:14:07 -05:00
Nick Sweeting	6189b1a8af	fix invalid old config vars	2019-01-12 03:13:55 -05:00
Nick Sweeting	b650c663a0	re-enable checking SSL by default	2019-01-11 22:48:09 -05:00
Nick Sweeting	f83750c545	cleanup options and make cli flags better for chrome headless timeouts	2019-01-11 22:38:50 -05:00
Nick Sweeting	911b84a79b	make wget cli options easier to read	2019-01-11 22:13:51 -05:00
Nick Sweeting	b77476cb83	fix user agent quoting	2019-01-11 19:54:43 -05:00
Nick Sweeting	33942d37cb	Update README.md	2019-01-11 07:57:46 -05:00
Nick Sweeting	4609b11d15	Update README.md	2019-01-11 07:56:24 -05:00
Nick Sweeting	5ea8a1ad9c	Update README.md	2019-01-11 07:21:30 -05:00
Nick Sweeting	e8808b0a1f	add WARC downloading	2019-01-11 07:02:49 -05:00
Nick Sweeting	329b06ec5a	make output dir in script	2019-01-11 06:53:34 -05:00
Nick Sweeting	74189966bf	add link to media	2019-01-11 06:50:42 -05:00
Nick Sweeting	a15a331798	fix media download with longer timeout	2019-01-11 06:33:35 -05:00
Nick Sweeting	c33f7ba91c	add ability to fetch media	2019-01-11 05:52:29 -05:00
Nick Sweeting	0df098717a	make git domains configurable	2019-01-11 05:27:25 -05:00
Nick Sweeting	d35c6cf8b5	fix fetch page title default	2019-01-11 05:19:22 -05:00
Nick Sweeting	67d103a293	dont float round timestamps	2019-01-11 05:19:07 -05:00
Nick Sweeting	827e15b31a	add git downloading	2019-01-11 05:18:49 -05:00
Nick Sweeting	5b6c768a47	autodetect common chrome binary locations	2019-01-11 04:34:16 -05:00
Nick Sweeting	8a13ca449d	Update README.md	2019-01-11 04:12:34 -05:00
Nick Sweeting	cf9d1875c7	add plain text link parsing	2019-01-11 04:09:39 -05:00
Nick Sweeting	7a9487fad9	Update README.md	2019-01-10 14:06:34 -05:00
Nick Sweeting	3c236a044b	Update README.md	2019-01-10 14:03:41 -05:00
Nick Sweeting	c0f46b2728	Set theme jekyll-theme-merlot	2019-01-10 14:00:41 -05:00
Nick Sweeting	b43a239a2e	Update README.md	2019-01-10 13:59:37 -05:00
Nick Sweeting	45e7bd5927	Update README.md	2019-01-10 13:51:14 -05:00
Nick Sweeting	0e72addcda	remove requests dependency	2019-01-09 04:05:34 -05:00
Nick Sweeting	84d4118e3c	Set theme jekyll-theme-minimal	2019-01-09 03:59:42 -05:00
Nick Sweeting	8cf1b04fed	install requests with pip	2019-01-05 15:26:49 -05:00
Nick Sweeting	0c37aa39c6	update to https	2018-12-31 22:31:36 -05:00
Nick Sweeting	60f8c95bf3	Update README.md	2018-12-31 21:25:22 -05:00
Nick Sweeting	6b48d881fa	Create CNAME	2018-12-31 21:18:34 -05:00