Nick Sweeting
a7d7644dca
Merge pull request #1205 from overhacked/fix_url_regex_hyphen
2023-08-09 14:20:07 -07:00
Ross Williams
c039ef05b3
Fix hyphen placement in util.URL_REGEX
...
Incorrect hyphen placement in `URL_REGEX` was allowing it to match more
characters than intended. In a regex character class, a literal hyphen
can only appear as the first character in the class, or it will be
interpreted as the delimiter of a range of characters.
The issue fixed here caused the range of characters from `[$-_]`
be treated as valid URL characters, instead of the intended set of three
characters `[-_$]`. The incorrect range interpretation inadvertantly
included most ASCII punctuation, most importantly the angle brackets,
square brackets, and single quote that the expression uses
to mark the end of a match.
This causes the expression to match a URL that has a "hostname" portion
beginning with one of the intended "stop parsing" characters. For
example:
```
https://<b>www</b>.example.com/ # MATCHES but should not
https://[for example] # MATCHES but should not
scheme='https://' # MATCHES, including final quote, but should not
```
Some test cases have been added to the `URL_REGEX` assert in
archivebox.parsers to cover this possibility.
2023-08-08 15:24:16 -04:00
Ross Williams
2076474252
Drop use of TypeAlias to maintain Python 3.9 compat
...
TypeAlias annotation was introduced in Python 3.10, and is not strictly
necessary. Drop use of it to maintain Python 3.9 compatibility.
2023-08-02 10:56:48 -04:00
Ross Williams
b44f7e68b1
Add URL-specific method allow/deny lists
...
Allows enabling only allow-listed extractors or disabling specific
deny-listed extractors for a regular expression matched against an added
site's URL.
2023-08-02 09:36:40 -04:00
Ross Williams
46e80dd509
Rename URL_(WHITE|BLACK)LIST to URL_(ALLOW|DENY)LIST
...
Retain aliases for old configuration files
2023-08-02 09:31:48 -04:00
Nick Sweeting
b773041952
Merge pull request #1199 from overhacked/chrome_version_detection_fix
2023-08-01 10:14:18 -07:00
Nick Sweeting
5b7ecfc872
Merge pull request #1197 from overhacked/bin_version_env_fix
2023-08-01 10:12:59 -07:00
Ross Williams
d0e65eba7f
More reliably detect Google Chrome version number
...
Previous method was splitting on the first whitespace, and missing the
version number when it appeared as `"Google Chrome 115.0.234.2342"`
instead of, i.e. `"Chromium 115.0.234.8283"`.
This commit changes the version detection to regex search for
whitespace, then one or more digits followed by a period, then at least
one more digit. Only the first sequence of digits is captured. Unless
Chrome radically changes their version numbering, this should capture
the first group of digits after the reported browser name, which would
be the major version.
2023-07-31 15:34:58 -04:00
Ross Williams
9d9872d325
bin_version means to modify, not replace environ
...
the `bin_version` function means to modify the environment,
not replace it entirely. Fixes bugs that occur when it wipes out the
PATH environment variable, such as when running in a virtual
environment.
2023-07-31 11:36:34 -04:00
Nick Sweeting
3e5e9c7a41
Merge pull request #1194 from wogong/patch-1
2023-07-29 09:29:34 -07:00
Zhen
3e9e221232
Fix Instapaper export link in README.md
...
Original link to Instapaper export `https://www.instapaper.com/user/export ` is broken: `405: Method Not Allowed`.
2023-07-28 07:58:58 +08:00
dependabot[bot]
0bf739b736
Bump word-wrap from 1.2.3 to 1.2.4
...
Bumps [word-wrap](https://github.com/jonschlinkert/word-wrap ) from 1.2.3 to 1.2.4.
- [Release notes](https://github.com/jonschlinkert/word-wrap/releases )
- [Commits](https://github.com/jonschlinkert/word-wrap/compare/1.2.3...1.2.4 )
---
updated-dependencies:
- dependency-name: word-wrap
dependency-type: indirect
...
Signed-off-by: dependabot[bot] <support@github.com>
2023-07-19 02:03:00 +00:00
Nick Sweeting
40ddd33602
Update README.md
2023-07-07 09:37:42 -07:00
mAAdhaTTah
181501fd36
Add Readwise Reader API parser
...
Implemented similar to the Pocket API.
2023-07-02 11:20:58 -04:00
Nick Sweeting
0d26538a4b
Update README.md example commands to use new docker compose format
2023-06-13 17:46:32 -07:00
Nick Sweeting
37d238cd31
Update README.md
2023-06-13 17:43:40 -07:00
Nick Sweeting
2580f76a2e
Update README.md
2023-06-13 17:41:08 -07:00
Nick Sweeting
571131d5f3
Update README.md to simplify intro instructions
2023-06-13 17:35:00 -07:00
Nick Sweeting
733dbfa1f3
Update scheduler to persist single shared contab via volume instead of requiring separate container for each job
2023-06-13 17:13:55 -07:00
Nick Sweeting
58d784cdd8
limit nginx config to only serve archive directory instead of main data folder root
2023-06-13 16:43:37 -07:00
Nick Sweeting
0e0b06bef1
Merge pull request #1159 from ArchiveBox/pirate-patch-1
2023-06-13 05:49:58 -07:00
Nick Sweeting
406e2b681d
Update docker-compose.yml scheduled task image and container name
2023-06-13 05:49:22 -07:00
Nick Sweeting
347b8d977d
Merge pull request #1154 from sissbruecker/fix/oneshot_exit_code
2023-05-30 17:34:25 -07:00
Sascha Ißbrücker
7bf4f40da0
just use out_dir
2023-05-29 10:03:49 +02:00
Sascha Ißbrücker
40c122515a
fix: make oneshot command return successful exist code
2023-05-29 10:01:27 +02:00
Nick Sweeting
32eef51081
Merge pull request #1153 from ArchiveBox/pirate-patch-1
...
Add links to new social media profiles under Further Reading
2023-05-27 06:08:25 -07:00
Nick Sweeting
d6780d1ab7
Add links to new social media profiles under Further Reading
2023-05-27 06:08:04 -07:00
Nick Sweeting
dbe1cd8dc0
Merge pull request #1152 from jess-sol/patch-1
2023-05-26 21:34:15 -07:00
J
aa5533b80f
Update export_browser_history.sh
...
Add tags to export of Firefox bookmarks. Ignore invalid bookmarks such as 'javascript:' or 'place:'
2023-05-25 09:08:13 -06:00
Nick Sweeting
081ba7e81b
Merge pull request #1147 from mrled/set-app-permissions-dockerfile
2023-05-06 00:20:50 -07:00
Micah R Ledbetter
3edb319ba9
Set /app permissions in the Dockerfile
...
Add a note in readme to be careful of permissions with --reload
2023-05-05 23:47:02 -05:00
Nick Sweeting
5bbc5228cc
Merge pull request #1146 from mrled/add-favicon-provider
2023-05-05 19:00:15 -07:00
Micah R Ledbetter
1e50ca243e
Add FAVICON_PROVIDER option for custom favicon service
2023-05-05 20:42:36 -05:00
Nick Sweeting
a1e2fce405
Update debian CI builds to use ubuntu 22.04 and actions/checkout@v3
2023-04-18 22:17:58 -07:00
Nick Sweeting
32fac5999e
Merge pull request #1131 from dcalano/dev
2023-03-30 13:14:17 -06:00
David Calano
f48e48e6da
Fix for Issue #1008
...
- Added missing decode() when setting pkg_path variable
2023-03-29 01:48:12 -04:00
Nick Sweeting
f86891bc33
Merge pull request #1128 from tejr/dev
2023-03-26 18:46:06 -06:00
Tom Ryder
53af810ff8
Add missing closing quote to style attribute
2023-03-27 10:54:04 +13:00
Nick Sweeting
829e377890
Merge pull request #1123 from fa0311/dev
2023-03-18 08:07:23 -06:00
ふぁ
44a5a5ed7e
add explicitly specify --headless=new
...
Signed-off-by: ふぁ <yuki@yuki0311.com>
2023-03-17 19:30:14 +09:00
Nick Sweeting
f5e631dbfa
Merge pull request #1120 from fa0311/dev
2023-03-15 17:05:46 -06:00
Nick Sweeting
9f42a3bf29
fix whitespace
2023-03-15 16:01:02 -07:00
ふぁ
d77c770c47
add CHROME_TIMEOUT args
...
Signed-off-by: ふぁ <yuki@yuki0311.com>
2023-03-14 20:29:41 +09:00
Nick Sweeting
cc53eceda2
recreate dockerx builder if missing platforms
2023-03-13 06:55:28 -06:00
Nick Sweeting
e5eee4e0b6
add cross-platform docker build support
2023-03-13 11:57:20 +00:00
Nick Sweeting
606fa397a4
disable passing timeout arg to chrome because v111 is crashing when passed
2023-03-13 10:50:18 +00:00
Nick Sweeting
1f1c70a8b1
remove --single-process from chrome args and add some rendering optimization args
2023-03-13 10:49:57 +00:00
Nick Sweeting
9599845b56
ensure DOM HTML dump is non-zero length file when retrying
2023-03-13 10:49:26 +00:00
Nick Sweeting
f94e8cd9b2
bump docker base image version and node version
2023-03-13 10:49:02 +00:00
Nick Sweeting
d72b1495d0
Merge pull request #1115 from fa0311/dev
2023-03-08 23:30:51 -08:00
