Sources / Press and newsletters
Sites that turn away AI crawlers
1 record was read from this source, dated 19 Nov 2026.
Its permanent copy in the Wayback Machine, 30 Sep 2026
Last checked 19 Nov 2026
Hold your company's records yourself? AIData.Broker values yours free against every public price on record, this source's included, and introduces it to every buyer that takes it, at no cost to you: the buyers pay us. Where it earns more, we run the sale ourselves, for a fee only out of a sale.
- No data upload
- About two minutes
- No email needed
What was read from it
| Record | Kind | As stated | Dated |
|---|---|---|---|
| PBS KIDS: Its robots.txt turns away 10 AI crawlers, GPTBot, ClaudeBot and CCBot among them | Closure signal | Turns away AI crawlers | 19 Nov 2026 |