Records / AI crawler census sweeps
AI crawler census, 2026-10-17: the robots.txt of 4,000 of the most visited sites (part 44 of 46 of the Tranco list of 2026-10-15)
219 of 1,898 sites that answered shut at least one AI crawler out
Dated 17 Oct 2026Source checked 17 Oct 2026Basis: observed
Where it was read
What it is about
Figures it counts in
- 300,000Most visited sites the census reads
- 4,000Robots.txt files the census reads each morning
- 11.4%Of them shut at least one AI crawler out of the whole site
- 8.4%Shut out two or more, one of them a training crawler
- 7.5%Of sites shut out GPTBot (OpenAI's training crawler)
- 6.0%Of sites shut out ClaudeBot (Anthropic's crawler)