Sources / Press and newsletters

Sites that turn away AI crawlers

1 record was read from this source, dated 19 Nov 2026.

Hold your company's records yourself? AIData.Broker values yours free against every public price on record, this source's included, and introduces it to every buyer that takes it, at no cost to you: the buyers pay us. Where it earns more, we run the sale ourselves, for a fee only out of a sale.

  • No data upload
  • About two minutes
  • No email needed

What was read from it

Records read from this source
RecordKindAs statedDated
PBS KIDS: Its robots.txt turns away 10 AI crawlers, GPTBot, ClaudeBot and CCBot among themClosure signalTurns away AI crawlers19 Nov 2026
Sites that turn away AI crawlers: what was read from it