Articles on ai
Every GhostPane article about ai — 8 in total.
-
Which AI crawlers are reading your site, and how to find out
GPTBot, ClaudeBot and a dozen others are fetching your pages right now. None run JavaScript, which is why your analytics shows none of it. Here is where the evidence actually lives.
-
llms.txt: what it proposes, and whether it does anything yet
A Markdown file at your domain root describing your site to language models. The idea is sound. Adoption is close to zero, and it is worth knowing which of those two facts should drive your decision.
-
A practical robots.txt for AI crawlers
A copy-and-paste file is the easy part. The two rules that catch people out are that named groups replace the wildcard, and that robots.txt is a request rather than an access control.
-
Why AI Overviews traffic is invisible, and what to do instead
Google's AI Overviews send clicks that look exactly like ordinary organic clicks. No tool can separate them, because the signal does not exist. Here is what you can actually infer.
-
Crawled but not cited: measuring what AI takes and what it returns
An AI crawler reading a page is a cost. A visitor arriving from an assistant is a return. Almost nobody measures both, which means almost nobody knows whether the exchange is fair.
-
Should you block AI crawlers? An honest cost-benefit
The argument is usually made on principle, in both directions. It is a business decision with measurable inputs, and the right answer is different for a documentation site than for a newspaper.
-
How to track traffic from ChatGPT, Perplexity and Claude
AI assistants send real visitors, and most analytics files them under the wrong channel or loses them entirely. Two configuration details decide whether the number you see is right.
-
GPTBot, ClaudeBot, PerplexityBot: what each one actually does
Not all AI crawlers do the same job. Some collect training data, some fetch a page because a person just asked a question. Blocking the wrong one costs you readers.