The [RFC](http://www.robotstxt.org/orig.html) mentions only `Disallow`
directive, so it must appear in the file.
`Allow` is an ad hoc agreement between search engines that no all of
them follow.
* Correct Windows paths
\\ is only for the beginning of UNC paths, e.g.
\\host-name\share-name\file-path
as per
https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file
* Remove spaces path list using brace expansion
In the shell, spaces are significant in a path with brace expansion, and
break it. (As opposed to in Rust module brace expansion.)
* add external_level and internal_level
* remove unnecessary debug derive on LinkDef
* clarify doc comment about link check levels
* simplify link checker logging
* add missing warn prefix
* simplify link level logging, remove "Level" from linklevel variants
* remove link level config from test site
* switch back to using bail! from get_link_domain
* move console's deps to libs
* remove unnecessary reference
* calling console::error/warn directly
* emit one error, or one warning, per link checker run
* various link checker level changes
* add docs about link checker levels
* remove accidentally committed test site
* remove completed TODO
Having to change the base URL to whatever ngrok URL was provisioned for
me was frustrating. This patch enables setting it to `/`, which will
then make the `get_url()` function simply return `/`.
* Add YAML to formats supported by load_data()
A fairly trivial addition; JSON and YAML are handled so similarly
that this was a matter of copying the JSON-relevant handlers and
editing the copies to handle YAML as well. The test file was
literally generated with 'json2yaml'.
The documentation has been updated to indicate that load_data() now
handles YAML code.
The CHANGELOG has been updated as well.
* After checking, I found that it's generally agreed the mime type is still application/x-yaml.
* Update comment, unify library importing.
I noticed one more place where the list of formats was supported,
and added YAML to that list.
I noticed that there's a singular place to load the `libs::` crate,
and unified by importing of serde_yaml in that place.
