98 commits

Author	SHA1	Message	Date
Angel Rey	a8a8fd14ac	Fixed indent headers.json	2020-09-23 11:07:00 -05:00
Angel Rey	852e3c9cff	Added headers extractor	2020-09-23 11:07:00 -05:00
Cristian	b18bbf8874	test: Fix tests post-rebase	2020-09-17 09:09:52 -05:00
apkallum	008769d296	add support for Paths in json encoder	2020-09-17 09:09:52 -05:00
Nick Sweeting	3658153cf8	fix url parsing through quotes	2020-08-18 08:04:57 -04:00
Cristian	d0d2991c69	fix: Change import that was not working	2020-07-31 12:15:00 -05:00
Cristian	6006b4f93b	refactor: Organize code to remove flake8 issues	2020-07-24 12:25:25 -05:00
Cristian	949f78aa65	fix: Use w3lib to improve the encoding extraction	2020-07-22 10:24:08 -05:00
Nick Sweeting	8cb530230c	fix docker SHM limited to 64mb chrome crash	2020-07-21 23:39:21 -04:00
apkallum	b7785c4138	use dateparser for parsing, let it handle error	2020-07-16 19:38:38 -04:00
Nick Sweeting	dfb83b4f27	add AttributeDict	2020-07-13 11:24:49 -04:00
Cristian	528fc8f1f6	fix: Improve encoding detection for rss+xml content types	2020-07-02 12:11:23 -05:00
Nick Sweeting	3ec97e5528	fix git conflict commited by accident	2020-07-02 03:22:37 -04:00
Nick Sweeting	8840ad72bb	remove circular import possibilities	2020-07-02 03:13:35 -04:00
Cristian	c971e00c9c	feat: Add stdout from process to the template	2020-07-01 12:23:59 -05:00
Nick Sweeting	c415420f33	improve sort columns and UI placeholders	2020-06-30 06:41:48 -04:00
Nick Sweeting	9f440c2cf8	use requests.get to fetch and decode instead of urllib	2020-06-30 05:55:54 -04:00
Nick Sweeting	cb67b09f9d	Merge branch 'master' into django	2020-06-25 21:30:29 -04:00
michael.bub	c79ce2b1f5	guess encoding via chardet if available	2020-02-15 13:58:07 +01:00
Mashiat Sarker Shakkhar	0bb216ce02	util.py: Use dateparser to parse date strings.	2019-09-10 23:51:09 -04:00
Nick Sweeting	500534f4be	fix missing comma in staticfile extensions list	2019-05-02 15:17:16 -04:00
Nick Sweeting	95007d9137	split up utils into separate files	2019-04-30 23:13:04 -04:00
Nick Sweeting	1b8abc0961	move everything out of legacy folder	2019-04-27 17:26:24 -04:00
Drewry Pope	332a32f4f9	Resolve 3 typos in util.py	2019-04-20 02:59:44 -05:00
Nick Sweeting	27708152d2	wip initial django setup	2019-04-02 16:36:41 -04:00
Nick Sweeting	f1075f2c7d	fix links index	2019-03-30 23:43:53 -04:00
Nick Sweeting	97249a1861	handle urls with special characters properly	2019-03-30 22:25:10 -04:00
Nick Sweeting	6a8f6f52af	0 mypy errors	2019-03-30 21:29:16 -04:00
Nick Sweeting	ac9fed06fd	whitespace	2019-03-30 17:59:56 -04:00
Nick Sweeting	880b425df6	add note about saving timestamp strings independently	2019-03-30 17:57:39 -04:00
Nick Sweeting	6d92784dbb	add docstring about timestamp parsing	2019-03-30 17:51:23 -04:00
Nick Sweeting	fffeb21ad4	remove progress dots in title fetching	2019-03-30 15:50:32 -04:00
Nick Sweeting	73f46b0b29	add proper typechecked json parsing and dumping	2019-03-30 15:03:46 -04:00
Nick Sweeting	4c8e45b8d7	save all imports to sources dir	2019-03-27 20:48:41 -04:00
Nick Sweeting	cc3d1e9cc9	limit length of stringified arg_vals in exceptions	2019-03-27 18:26:22 -04:00
Nick Sweeting	d2a34f2602	switch to atomic disk writes for all index operations	2019-03-27 18:24:57 -04:00
Nick Sweeting	a214bd7c02	make everything take link_dir as an optional arg since its derivable from link url	2019-03-27 18:24:30 -04:00
Nick Sweeting	93216a3c3e	new version handling and absolute imports	2019-03-27 15:35:13 -04:00
Nick Sweeting	002206d4fe	remove distutils in favor of shutil	2019-03-27 11:39:51 -04:00
Nick Sweeting	2ed91fe429	new details page design	2019-03-27 03:49:39 -04:00
Nick Sweeting	c9c5b04df0	full type-hinting coverage	2019-03-26 23:25:07 -04:00
Nick Sweeting	ab09560f14	working runtime type casting and enforcement for a wide range of types	2019-03-26 22:26:21 -04:00
Nick Sweeting	25a107df43	switch to dataclasses, working Link type hints everywhere	2019-03-26 19:21:34 -04:00
Nick Sweeting	76abc58135	switch to strict type hints with NamedTuples instead of dicts	2019-03-26 05:33:34 -04:00
Nick Sweeting	4f8c99011a	fix terminal resizing making progress bar go crazy	2019-03-26 05:30:23 -04:00
Nick Sweeting	63abc8980c	add mypy type hints	2019-03-26 03:20:41 -04:00
Nick Sweeting	f7a0568a6c	better chrome options loading	2019-03-22 23:00:53 -04:00
Nick Sweeting	4c499d77b6	move dependency checking into config file	2019-03-22 22:05:45 -04:00
Nick Sweeting	d06775923b	move latest to derived data using history	2019-03-22 21:38:08 -04:00
Nick Sweeting	69f837bbf6	simplify chrome_user_data_dir default	2019-03-22 21:37:02 -04:00