Commit graph

106 commits

Author SHA1 Message Date
Tom Parker-Shemilt
4b7e6d1488 Get handle once per recursive sequence of get_url calls 2020-06-03 17:57:07 +01:00
Tom Parker-Shemilt
5e9357ed6f Swap back to older reqwest API 2020-06-02 23:02:18 +01:00
Tom Parker-Shemilt
87ad03cb15 Remove handle debug because it overloads travis 2020-06-02 22:40:28 +01:00
Tom Parker-Shemilt
a1abde30b9 Upgrade packages and change over deprecated Reqwest function 2020-06-02 22:32:16 +01:00
Tom Parker-Shemilt
c680c63f8a Disable connection pooling, as it seems to break stuff 2020-06-02 22:25:01 +01:00
Tom Parker-Shemilt
25084b928a Various debug for checks 2020-06-02 22:24:49 +01:00
Tom Parker-Shemilt
1f6cb3749b Panic earlier on 404'ed links 2020-05-01 00:22:48 +01:00
Tom Parker-Shemilt
6532f7681f Only do Github API rewrite if have environment variables 2020-04-30 22:04:01 +01:00
Tom Parker-Shemilt
2dfb07854f
Merge pull request #867 from palfrey/add-github-secrets
Support optional use of Github secrets to workaround rate limits
2020-04-25 18:26:03 +01:00
Tom Parker-Shemilt
1efa0de042 Missed two other cases of needing to print url with link 2020-04-21 11:18:53 +01:00
Tom Parker-Shemilt
e953dbb269 Print url before Link 2020-04-21 11:04:59 +01:00
Tom Parker-Shemilt
0e3bada631 Support optional use of Github secrets to workaround rate limits (fixes #859) 2020-04-03 22:10:09 +01:00
Tom Parker-Shemilt
15dbb3602b For most errors, intermittent failures are ok, not so much for redirects 2020-04-03 21:12:54 +01:00
Tom Parker-Shemilt
60e614f51e Nicer "failure only occured X ago" message 2020-03-26 23:29:34 +00:00
Tom Parker-Shemilt
d4b44fd6d5 Don't remove last_working if a link fails 2020-03-26 23:22:40 +00:00
Tom Parker-Shemilt
12464c2844 Redo (not-)working with a enum, not a pair of variables 2020-03-10 21:17:59 +00:00
Tom Parker-Shemilt
592a4bed92 Output all the info on link failure 2020-03-08 16:49:14 +00:00
Tom Parker-Shemilt
edc17508e1 Only write every so often 2020-03-08 16:49:04 +00:00
Tom Parker-Shemilt
2901a666cb Merge branch 'master' into cache-results 2020-03-03 23:10:02 +00:00
Tom Parker-Shemilt
f3ac7ba5d3
Merge pull request #840 from palfrey/move-cleanup
Move cleanup into bin folder
2020-03-03 23:08:37 +00:00
Tom Parker-Shemilt
0d798c3896 Github workflows have started to do 406, unless we add an explicit image to the Accept header 2020-02-26 21:26:18 +00:00
Tom Parker-Shemilt
43eab5fa9a Merge branch 'master' into better-unknown-travis-detection 2020-02-25 21:26:29 +00:00
Tom Parker-Shemilt
9d41a10148 Cache the results of getting links across Travis build and only re-check
every so often
2020-02-24 23:41:45 +00:00
Tom Parker-Shemilt
d649b575a7 Update cleanup instructions 2020-02-24 22:38:19 +00:00
Tom Parker-Shemilt
e6520cbac0 Move cleanup to bin folder and run it on the repo 2020-02-24 22:36:50 +00:00
Tom Parker-Shemilt
ecd6658ec6 Improve detection of "unknown" travis builds 2020-02-16 20:44:45 +00:00
Tom Parker-Shemilt
0a527ff6e3 Catch no-branch cases for Travis 2020-02-12 23:38:26 +00:00
Tom Parker-Shemilt
5bceef3656 Catch github actions badges with no branch 2020-02-12 23:32:02 +00:00
Tom Parker-Shemilt
89efda92e3 Azure devops does 302 twice, so need to cope with that 2020-02-10 21:02:28 +00:00
Tom Parker-Shemilt
4d048eba27 Add workaround for Youtube 302s (fixes #814) 2020-01-30 15:23:42 +00:00
Tom Parker-Shemilt
dec99f4416 Also look for travis.org as well as .com 2020-01-20 18:05:20 +00:00
Tom Parker-Shemilt
f0af98f3fc Detect unknown status travis builds 2020-01-20 17:53:24 +00:00
Tom Parker-Shemilt
7ea1063c32 Update user-agent so Reddit stops complaining 2020-01-20 13:50:02 +00:00
Tom Parker-Shemilt
6cb9433de0 Look for links/images in HTML content in the Markdown 2020-01-18 14:01:05 +00:00
Tom Parker-Shemilt
a37b1cfee3 Redo error checking with failure and less verbose formatting 2020-01-17 23:24:03 +00:00
Tom Parker-Shemilt
5733288c9a Record actual real errors 2020-01-13 22:21:03 +00:00
Tom Parker-Shemilt
f799381f75 Add 20 second timeout 2020-01-13 22:13:13 +00:00
Tom Parker-Shemilt
c25072ec06 Add workaround for github actions 2020-01-13 22:13:00 +00:00
Tom Parker-Shemilt
ff703bac03 Fix accept header so we also get github svgs 2020-01-13 21:53:25 +00:00
Tom Parker-Shemilt
a6b79d73ec Also pick up image tags 2020-01-13 17:04:46 +00:00
Tom Parker-Shemilt
fd843a5dd8 Add extra debug on redirects 2020-01-13 16:17:24 +00:00
Tom Parker-Shemilt
39b2e4b3b6 Add text/hmtl header to fix crates.io 2020-01-13 16:07:48 +00:00
Tom Parker-Shemilt
70361602dd Detect redirects 2020-01-13 08:39:38 +00:00
Tom Parker-Shemilt
67bfa13da5 Retry up to 5 times just in case on failure 2020-01-11 14:45:49 +00:00
Tom Parker-Shemilt
f6e1302d17 Limit running gets to 20 to stop issues with file handles 2020-01-11 12:06:40 +00:00
Tom Parker-Shemilt
57a7f96ba5 Start writing a rust-based url checker 2020-01-10 23:07:36 +00:00
kud1ing
0d11f0b0c5 remove 2017-05-03 18:46:46 +02:00
kud1ing
4b22d82e49 merge the deployment sections 2017-04-29 10:00:25 +02:00
kud1ing
2978c44acc add section 'Text search' 2016-12-28 17:32:40 +01:00
kud1ing
d0d937e00a collect the footer Markdown events 2016-12-10 21:49:04 +01:00
kud1ing
daae790834 flesh out the model some more 2016-12-10 20:59:18 +01:00
kud1ing
76bbf8d6d2 refactor 2016-12-10 20:47:16 +01:00
kud1ing
b9996a01cd cleanup 2016-12-10 20:38:31 +01:00
kud1ing
98d01abf50 more high-level parsing 2016-12-10 20:32:42 +01:00
kud1ing
21f8b53241 cleanup 2016-12-10 19:23:48 +01:00
kud1ing
eb15075946 add a parser stub for the README.md file 2016-12-10 18:07:07 +01:00