robots-txt
Articles on robots-txt
Every GhostPane article about robots-txt — 2 in total.
-
A practical robots.txt for AI crawlers
A copy-and-paste file is the easy part. The two rules that catch people out are that named groups replace the wildcard, and that robots.txt is a request rather than an access control.
-
GPTBot, ClaudeBot, PerplexityBot: what each one actually does
Not all AI crawlers do the same job. Some collect training data, some fetch a page because a person just asked a question. Blocking the wrong one costs you readers.