Mirrors/gotosocial
2
0
Fork 0
mirror of https://github.com/superseriousbusiness/gotosocial synced 2024-11-10 06:54:16 +00:00
Code Issues Projects Releases Packages Wiki Activity

[chore] Update robots.txt with more AI scrapers (#3248)

Browse source
This commit is contained in:
Daenney 2024-08-29 17:42:48 +02:00 committed by GitHub
parent 1f3dfbf10c
commit 277b043633
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
1 changed files with 5 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

5
internal/web/robots.go
View file

@ -31,6 +31,8 @@ const (
# AI scrapers and the like.
# https://github.com/ai-robots-txt/ai.robots.txt/
User-agent: AI2Bot
User-agent: Ai2Bot-Dolma
User-agent: AdsBot-Google
User-agent: Amazonbot
User-agent: anthropic-ai
@ -50,9 +52,11 @@ User-agent: GoogleOther
User-agent: GoogleOther-Image
User-agent: GoogleOther-Video
User-agent: GPTBot
User-agent: ICC-Crawler
User-agent: ImagesiftBot
User-agent: img2dataset
User-agent: Meta-ExternalAgent
User-agent: Meta-ExternalFetcher
User-agent: OAI-SearchBot
User-agent: omgili
User-agent: omgilibot
@ -61,6 +65,7 @@ User-agent: PetalBot
User-agent: Scrapy
User-agent: Timpibot
User-agent: VelenPublicWebCrawler
User-agent: Webzio-Extended
User-agent: YouBot
Disallow: /