Records / Laws and rulings

EDPB Opinion 28/2024, para 105(c): honouring robots.txt as a measure for web scraping (EDPB Opinion 28/2024, adopted 17 December 2024, para 105(c))

Under EU data protection law, honouring robots.txt is one of the measures that can help a scraper of personal data justify the collection, so in Europe robots.txt carries legal weight beyond notice.

Dated 28 Oct 2026Source checked 28 Oct 2026Quote found on its page again 30 Oct 2026Basis: primary text

Selling a company's data? This is one of the rules a sale has to meet, and the guide beside the button reads it in plain words.

Value your dataIs It Legal to Scrape Data for AI Training? (US Law, 2026)
  • No data upload
  • About two minutes
  • No email needed
In the source's own words
“Specific measures, when not necessary under the second step of the legitimate interest assessment, may prove useful to mitigate the risk in the context of web scraping. These may include technical measures, such as: ... Excluding collection from websites (or sections of websites) which clearly object to web scraping and the reuse of their content for the purpose of building AI training databases (for example, by respecting robots.txt or ai.txt files or any other recognised mechanism to express exclusion from automated crawling or scraping).”
Also read from the same source

All 8 records from this source

Figures it counts in

It is shown on its own, not counted into a figure.

EDPB Opinion 28/2024, para 105(c): honouring robots.txt as a measure for web scraping… | AIData.Broker