mirror of
https://github.com/superseriousbusiness/gotosocial
synced 2024-11-26 22:30:25 +00:00
[feature] Block Amazonbot (#2692)
Blocks the Amazon crawler bot. Closes: #2686
This commit is contained in:
parent
9cadc764b3
commit
6528592dd2
1 changed files with 5 additions and 0 deletions
|
@ -64,6 +64,11 @@ Disallow: /
|
|||
User-agent: WellKnownBot
|
||||
Disallow: /
|
||||
|
||||
# Block Amazonbot, because Amazon.
|
||||
# https://developer.amazon.com/amazonbot
|
||||
User-agent: Amazonbot
|
||||
Disallow: /
|
||||
|
||||
# Rules for everything else.
|
||||
User-agent: *
|
||||
Crawl-delay: 500
|
||||
|
|
Loading…
Reference in a new issue