1419 commits

Author	SHA1	Message	Date
Nick Sweeting	af669d2f37	rename api files for clarity	2024-04-25 05:55:47 -07:00
Nick Sweeting	75153252dc	big overhaul of REST API, split into auth, core, and cli methods	2024-04-25 03:56:22 -07:00
Nick Sweeting	e5aba0dc2e	fix urljoin bug causing multiple slashes to be merged into one	2024-04-24 19:41:31 -07:00
Nick Sweeting	6cb357e76c	fix fix_url_from_markdown assertion to be valid url	2024-04-24 19:41:11 -07:00
Nick Sweeting	128419f991	expand comment about markdown url trailing paren trimming	2024-04-24 17:50:18 -07:00
Nick Sweeting	beb3932d80	replace uses of URL_REGEX with find_all_urls to handle markdown better	2024-04-24 17:45:45 -07:00
Nick Sweeting	98c5e69203	bump lockfiles	2024-04-24 14:38:21 -07:00
Nick Sweeting	8dcfa93ec6	Merge branch 'main' into dev	2024-04-24 14:32:07 -07:00
Nick Sweeting	17f40f3ada	Merge branch 'dev' into fix-URL_REGEX	2024-04-23 19:53:58 -07:00
Nick Sweeting	c6f8a33a63	Update util.py	2024-04-23 19:53:18 -07:00
Nick Sweeting	63fc317229	minor pylint fixes in logging_util	2024-04-23 17:45:36 -07:00
Nick Sweeting	756e159dfe	add new bin/lock_pkgs.sh to generate package lockfiles consistently	2024-04-23 17:45:36 -07:00
longzai	e4dc2701ef	fix URL_REGEX 2	2024-04-11 15:51:55 +08:00
Brandl	5f9aac18f2	api v1	2024-04-10 01:29:24 +02:00
longzai	4ae765ec27	fix the URL_REGEX used in generic_html parsers ... Signed-off-by: longzai <437172242@qq.com>	2024-04-08 04:53:05 +08:00
Nick Sweeting	e48159b8a0	cleanup docker-compose by storing crontabs in data dir	2024-03-26 15:24:05 -07:00
Nick Sweeting	ac73fb5129	merge fixes	2024-03-26 15:22:40 -07:00
Nick Sweeting	b4c3aa5097	Merge branch 'main' into dev	2024-03-26 15:01:36 -07:00
Nick Sweeting	a4453b6f87	fix PERSONAS PERSONAS_DIR typo	2024-03-26 14:19:25 -07:00
jim winstead	8b1b01e508	Update to Django 4.2.x, now in LTS until April 2026	2024-03-25 17:46:01 -07:00
Nick Sweeting	c0b5dbcecb	create new data/personas dir to hold cookies and chrome profiles	2024-03-18 14:41:39 -07:00
Nick Sweeting	c5bb99dce1	explicitly use Default profile inside user data dir	2024-03-18 14:40:40 -07:00
Nick Sweeting	1fc5d7c5c8	add USER_AGENT config option to set all USER_AGENTs at once	2024-03-18 14:39:09 -07:00
jim winstead	5478d13d52	Add generic_jsonl parser ... Resolves #1369	2024-03-14 15:42:29 -07:00
Nick Sweeting	ca2c484a8e	Add _EXTRA_ARGS for various extractors (#1360)	2024-03-14 01:55:09 -07:00
Nick Sweeting	48f4b12ae2	Use COOKIES_FILE to fetch page titles (#1364)	2024-03-14 01:52:44 -07:00
Nick Sweeting	099f7d00fe	Use feedparser for RSS parsing (#1362) ... Fixes #1171 Fixes #870 (probably, would need to test against a Wallabag Atom file to Fixes #135 Fixes #123 Fixes #106	2024-03-14 01:51:45 -07:00
Nick Sweeting	3512dc7e60	Disable searching for existing chrome user profiles by default	2024-03-14 00:58:45 -07:00
Ben Muthalaly	f4deb97f59	Add ARGS and EXTRA_ARGS for Mercury extractor	2024-03-05 21:15:38 -06:00
Ben Muthalaly	d8cf09c21e	Remove unnecessary variable length args for dedupe	2024-03-05 21:13:45 -06:00
Ben Muthalaly	5082d61613	Merge branch 'title-cookies-file' of https://github.com/benmuth/ArchiveBox into title-cookies-file	2024-03-05 02:03:03 -06:00
Ben Muthalaly	4686da91e6	Fix cookies being set incorrectly	2024-03-05 01:48:35 -06:00
Naomi Phillips	a729480b75	Add COOKIES_FILE support for singlefile extractor	2024-03-03 02:32:46 -05:00
Ben Muthalaly	d74ddd42ae	Flip dedupe precedence order	2024-03-01 14:50:32 -06:00
jim winstead	9f462a87a8	Use feedparser for RSS parsing in generic_rss and pinboard_rss parsers ... The feedparser packages has 20 years of history and is very good at parsing RSS and Atom, so use that instead of ad-hoc regex and XML parsing. The medium_rss and shaarli_rss parsers weren't touched because they are probably unnecessary. (The special parse for pinboard is just needing because of how tags work.) Doesn't include tests because I haven't figured out how to run them in the docker development setup. Fixes #1171	2024-03-01 11:25:45 -08:00
Nick Sweeting	a577d1ed23	Merge branch 'dev' into title-cookies-file	2024-02-29 21:29:36 -08:00
Nick Sweeting	fe11e1c2f4	check if COOKIE_FILE is file	2024-02-28 18:19:44 -08:00
jim winstead	ccabda4c7d	Handle list of tags in JSON, and be more clever about comma vs. space	2024-02-28 17:38:49 -08:00
jim winstead	178e676e0f	Fix JSON parser by not always mangling the input ... Rather than by assuming the JSON file we are parsing has junk at the beginning (which maybe only used to happen?), try parsing it as-is first, and then fall back to trying again after skipping the first line Fixes #1347	2024-02-27 14:48:19 -08:00
Ben Muthalaly	68326a60ee	Add cookies file to http request in download_url	2024-02-27 15:30:31 -06:00
Ben Muthalaly	4d9c5a7b4b	Add CHROME_EXTRA_ARGS ... Also fix `YOUTUBEDL_EXTRA_ARGS`.	2024-02-23 18:40:03 -06:00
Ben Muthalaly	ab8f395e0a	Add YOUTUBEDL_EXTRA_ARGS	2024-02-23 15:40:31 -06:00
Ben Muthalaly	4e69d2c9e1	Add EXTRA_*_ARGS for wget, curl, and singlefile	2024-02-22 23:04:11 -06:00
Nick Sweeting	8b9bc3dec8	minor fixes	2024-02-22 04:50:22 -08:00
Nick Sweeting	6a4e568d1b	new archivebox update speed improvements	2024-02-22 04:50:22 -08:00
Nick Sweeting	903c72fa88	fix typing errors	2024-02-12 21:28:52 -08:00
Nick Sweeting	91c4641199	skip dir size calculation when path is too long	2024-02-12 21:26:34 -08:00
Nick Sweeting	19aefc85e6	fix get_system_user failing on uid 999 in k3s	2024-02-08 18:58:12 -08:00
Nick Sweeting	a4bd441077	Update config.py fix trim-filenames	2024-01-31 01:59:43 -08:00
Nick Sweeting	e8772513ca	Update archivebox/core/auth_ldap.py ... add log line	2024-01-24 11:00:42 -08:00