Articles on crawlers
Every GhostPane article about crawlers — 4 in total.
-
Which AI crawlers are reading your site, and how to find out
GPTBot, ClaudeBot and a dozen others are fetching your pages right now. None run JavaScript, which is why your analytics shows none of it. Here is where the evidence actually lives.
-
A practical robots.txt for AI crawlers
A copy-and-paste file is the easy part. The two rules that catch people out are that named groups replace the wildcard, and that robots.txt is a request rather than an access control.
-
Should you block AI crawlers? An honest cost-benefit
The argument is usually made on principle, in both directions. It is a business decision with measurable inputs, and the right answer is different for a documentation site than for a newspaper.
-
GPTBot, ClaudeBot, PerplexityBot: what each one actually does
Not all AI crawlers do the same job. Some collect training data, some fetch a page because a person just asked a question. Blocking the wrong one costs you readers.