Mirrors/ArchiveBox
2
0
Fork 0
mirror of https://github.com/ArchiveBox/ArchiveBox synced 2024-11-26 14:10:20 +00:00
Code Issues Projects Releases Packages Wiki Activity
712 commits 6 branches 78 tags 19 MiB
Commit graph

110 commits

Author SHA1 Message Date
Nick Sweeting
756f31a50b hide common youtubedl failures 2019-02-04 20:08:54 -08:00
Nick Sweeting
2b45792ad8 new test files 2019-02-04 20:04:03 -08:00
Nick Sweeting
265e1bd830 fix parsing links from stdin with whitespace 2019-02-04 20:03:32 -08:00
Nick Sweeting
794c043fa4 fix chrome data dir in wrong folder 2019-02-04 20:03:16 -08:00
Nick Sweeting
9f5441c4f3 quieter media downloads 2019-02-04 20:03:06 -08:00
Nick Sweeting
4e5b466bb2 fix broken wget user agents 2019-02-04 20:02:54 -08:00
Nick Sweeting
d8c7c98c01 fix exception format string 2019-02-04 20:02:40 -08:00
Nick Sweeting
705abda1a5 send ArchiveBox user agent to archive.org 2019-02-04 20:02:33 -08:00
Nick Sweeting
8a9b4d6975 dont git clone urls with hashes 2019-02-04 19:21:08 -08:00
Nick Sweeting
b8e5e1d5bd skip media when folder exists, even if empty 2019-02-04 19:21:00 -08:00
Nick Sweeting
901666bae6 dont show wget errors unless all files failed to download 2019-02-04 19:20:36 -08:00
Nick Sweeting
3ac0efb619 dont parse quotes as part of urls 2019-02-04 18:54:02 -08:00
Nick Sweeting
af8b9b5fdf use parsed timestamps instead of now 2019-02-04 18:53:54 -08:00
Nick Sweeting
21c62043e3 new logo 2019-02-04 18:45:46 -08:00
Nick Sweeting
543d2fc010 tweak index visual header style 2019-02-04 18:45:36 -08:00
Nick Sweeting
c969451a1c dont clobber pocket timestamps 2019-02-04 18:45:25 -08:00
Nick Sweeting
c32a385e8f show which format file was parsed as 2019-02-04 18:45:18 -08:00
Nick Sweeting
ae0c20dc76 fix flake8 errors 2019-02-04 17:52:53 -08:00
Nick Sweeting
eeb89ae2cc
Merge pull request #138 from luoliyan/improve-purge-script 
Improve index purge script
 2019-02-04 17:49:22 -08:00
luoliyan
fee4565194 Improve index purge script 2019-02-05 06:48:49 +09:30
Nick Sweeting
ad7038e031 catch json parse errors in link archiving 2019-02-04 08:00:08 -08:00
Nick Sweeting
c37941efd1 dont match closing tags in full text 2019-01-31 19:46:34 -08:00
Nick Sweeting
9eb79258bb check chrome version on startup if using chrome 2019-01-29 17:08:15 -08:00
Nick Sweeting
b53251fe45 fix parsers 2019-01-26 15:09:58 -08:00
Nick Sweeting
ff125d990c fix order of parsers not enforced on python 3.5 2019-01-25 21:02:25 -08:00
Nick Sweeting
e60070dbb2 add youtubedl to help str 2019-01-25 17:38:47 -08:00
Nick Sweeting
20de451515 add linux config example 2019-01-23 01:48:04 -05:00
Nick Sweeting
95301c9306 better default config 2019-01-23 01:42:55 -05:00
Nick Sweeting
2c30cd9468 better closure 2019-01-23 01:08:23 -05:00
Nick Sweeting
db0461ec4e add timeout to title fetching 2019-01-23 01:08:23 -05:00
Nick Sweeting
e1be96e597 working docker-compose with google chrome 2019-01-23 01:08:23 -05:00
Nick Sweeting
f25be8bc24 add chrome user agent example in config 2019-01-20 14:08:33 -05:00
Nick Sweeting
071b39b944 nicer error formats 2019-01-20 14:08:33 -05:00
Nick Sweeting
de6eb649e0 disable wget timestamping if using warcs 2019-01-20 14:08:33 -05:00
Nick Sweeting
ce07a6c738 patch subprocess.run to have better timeout handling 2019-01-20 14:08:33 -05:00
Nick Sweeting
e620722bb9 fix timeouts hanging the progressbar 2019-01-20 14:08:33 -05:00
Nick Sweeting
7ea36c4adb bump timeouts and improve curl archive method 2019-01-20 14:08:33 -05:00
Nick Sweeting
cc8611de83
Update config.py 2019-01-14 22:46:35 -05:00
Nick Sweeting
c42fcd42d7 fetch warc file inline with wget instead of as separate step 2019-01-14 22:43:14 -05:00
Nick Sweeting
300b5c6182 put ArchiveBox and wget version in user agent 2019-01-14 18:17:30 -05:00
Nick Sweeting
39f5e12364 allow passing import list via stdin 2019-01-14 18:11:48 -05:00
Nick Sweeting
cb60bad1d7 disable WARC by default 2019-01-12 20:19:17 -05:00
Nick Sweeting
321cdeda11 add mirror to git clone 2019-01-12 20:13:13 -05:00
Nick Sweeting
6189b1a8af fix invalid old config vars 2019-01-12 03:13:55 -05:00
Nick Sweeting
b650c663a0 re-enable checking SSL by default 2019-01-11 22:48:09 -05:00
Nick Sweeting
f83750c545 cleanup options and make cli flags better for chrome headless timeouts 2019-01-11 22:38:50 -05:00
Nick Sweeting
911b84a79b make wget cli options easier to read 2019-01-11 22:13:51 -05:00
Nick Sweeting
b77476cb83 fix user agent quoting 2019-01-11 19:54:43 -05:00
Nick Sweeting
e8808b0a1f add WARC downloading 2019-01-11 07:02:49 -05:00
Nick Sweeting
74189966bf add link to media 2019-01-11 06:50:42 -05:00
Nick Sweeting
a15a331798 fix media download with longer timeout 2019-01-11 06:33:35 -05:00
Nick Sweeting
c33f7ba91c add ability to fetch media 2019-01-11 05:52:29 -05:00
Nick Sweeting
0df098717a make git domains configurable 2019-01-11 05:27:25 -05:00
Nick Sweeting
d35c6cf8b5 fix fetch page title default 2019-01-11 05:19:22 -05:00
Nick Sweeting
67d103a293 dont float round timestamps 2019-01-11 05:19:07 -05:00
Nick Sweeting
827e15b31a add git downloading 2019-01-11 05:18:49 -05:00
Nick Sweeting
5b6c768a47 autodetect common chrome binary locations 2019-01-11 04:34:16 -05:00
Nick Sweeting
cf9d1875c7 add plain text link parsing 2019-01-11 04:09:39 -05:00
Nick Sweeting
0e72addcda remove requests dependency 2019-01-09 04:05:34 -05:00
Nick Sweeting
57d42339a4 rename pip dir archive to archivebox 2018-12-31 20:53:01 -05:00