Is It Legal to Scrape Data for AI Training? (US Law, 2026)

Scraping public web pages to train AI is usually not a federal crime in the US, but it is rarely free of legal risk, and data obtained that way is hard to sell on. Since the Supreme Court's decision in Van Buren v. United States (2021) and the Ninth Circuit's ruling in hiQ v. LinkedIn (2022), the federal anti-hacking law reaches getting past a gate, not reading pages anyone can see. What remains are four other bodies of law: contract (a site's terms, if you agreed to them), copyright (where fair use is unsettled), anti-circumvention (in July 2026 a federal court let Reddit's claims proceed against scrapers accused of getting around anti-bot barriers), and privacy law (scraping public photos led to a biometric privacy settlement paid in a 23% equity stake in Clearview).

This guide is for a company deciding whether to scrape, a buyer's data team deciding whether to accept scraped data, and a holder whose site is being scraped. It walks through each body of law in the order a scraper meets it, with the cases that define it today, explains what robots.txt does and does not do (with our own census of how many sites shut AI crawlers out), and ends with the commercial point that matters most: you cannot license rights you never had, which is why buyers increasingly pay for licensed data instead.

This guide is research from public sources, not legal advice.

Contents

1. Short answer: five bodies of law, not one
2. The Computer Fraud and Abuse Act after Van Buren and hiQ
3. Contracts and terms of service
4. Trespass to chattels
5. Copyright and fair use
6. Anti-circumvention: getting around technical barriers
7. Robots.txt: not law, but evidence of notice
8. Privacy law: Clearview and Illinois BIPA
9. The leading cases at a glance
10. Why scraped data is hard to resell
11. Questions people ask
12. What this means for you

1. Short answer: five bodies of law, not one

There is no single US statute on web scraping. Whether a particular scrape is lawful depends on how the data was reached (an open page, a logged-in account, a page behind a CAPTCHA), what was taken (facts, copyrighted text, faces), what you agreed to along the way, and what you do with the result (train a model, display it, sell it). Each of those questions belongs to a different body of law, and a scraper can be on safe ground under one and exposed under another. hiQ, for example, won its argument under the anti-hacking law and then lost on contract.

The table sets out the five, with the authority that defines each one today. The sections that follow take them in turn.

Anti-circumvention is part of copyright law (the DMCA amended the Copyright Act), but it works differently enough to get its own row, and a scraper often meets the two together. Our case ledger holds 83 lawsuits, rulings and settlements about training AI on data; several of the newest are about how the data was reached rather than whether training is fair.

2. The Computer Fraud and Abuse Act after Van Buren and hiQ

What the law says

The Computer Fraud and Abuse Act (CFAA) is the federal anti-hacking statute. It makes it a crime, and gives victims a civil claim, to access a computer "without authorization" or to "exceed authorized access", which the statute defines as using authorized access to obtain or alter information in the computer that the accesser is not entitled so to obtain or alter. For two decades, website operators argued that a scraper who ignored their terms of service, or kept going after a cease-and-desist letter, was accessing "without authorization". If that were right, ordinary scraping would be a federal crime.

Van Buren: gates up or gates down

In June 2021 the Supreme Court rejected the broad reading. Van Buren was a police officer who ran a licence-plate search, which he was allowed to do, for a corrupt purpose. The Court held the CFAA did not cover him: the "exceeds authorized access" clause reaches people who enter parts of a system that are off limits to them, and " It does not cover those who, like Van Buren, have improper motives for obtaining information that is otherwise available to them." The Court described the test as a gates-up-or-down inquiry: either you may enter a computer or an area of it, or you may not. It also warned that the broader reading would turn violations of website terms of service into crimes.

hiQ: public pages have no gates

hiQ Labs scraped public LinkedIn profiles to sell analytics to employers. LinkedIn sent a cease-and-desist letter and blocked it; hiQ sued and won a preliminary injunction letting it keep scraping. After Van Buren, the Supreme Court sent the case back, and in April 2022 the Ninth Circuit reached the same result. A public website, it said, has no gates to begin with: " applying the "gates" analogy to a computer hosting publicly available webpages, that computer has erected no gates to lift or lower in the first place." Scraping pages anyone can view without logging in is therefore unlikely to be access "without authorization", even after the owner objects.

Two limits matter. First, hiQ was a preliminary injunction ruling, decided on "serious questions", not a final judgment. Second, the court said in the same opinion that site operators are not without resort: trespass to chattels, copyright, misappropriation, unjust enrichment, breach of contract and privacy claims may all still apply.

What the CFAA still covers

The gate is the line. Scraping behind a login you are not entitled to, using fake accounts, reusing someone else's credentials, or continuing after your own access has been revoked can still be access without authorization. hiQ itself is the example: when the case ended, hiQ agreed that LinkedIn could establish CFAA liability based on hiQ's access to password-protected pages using fake accounts. States have their own computer crime laws (California's computer crime statute was pleaded in hiQ), and they do not all track Van Buren. For a scraper, the practical rule is simple: stay on the public side of every login.

3. Contracts and terms of service

Browsewrap and clickwrap

If the CFAA rarely reaches public scraping, a site's terms of service might. But terms only bind someone who agreed to them. The Ninth Circuit's description of the two kinds of online agreement is the starting point: " Contracts formed on the Internet come primarily in two flavors": clickwrap, where the user must click "I agree" after being shown the terms, and browsewrap, where the terms sit behind a link, usually at the bottom of the page, and the site claims that using it means agreeing. Courts enforce clickwrap readily. They are reluctant to enforce browsewrap, especially against individuals: the court in Nguyen held that the onus must be on website owners to put users on notice of the terms they want to enforce.

For scraping, that means a logged-out bot that never clicked anything is hard to bind by contract, while a scraper that opened an account and accepted the terms usually is.

Meta v. Bright Data and X Corp. v. Bright Data (2024)

Bright Data runs one of the largest web data businesses, selling scraping tools and scraped datasets, including from social platforms. Two platforms sued it in San Francisco in 2023, and both lost in 2024.

In Meta v. Bright Data, Judge Edward Chen held in January 2024 that Facebook's and Instagram's terms govern account holders using their accounts, not visitors. Bright Data had held accounts, but its scraping was done logged out, of public pages, and the undisputed evidence establishes that Bright Data only engaged in logged-out scraping of public data and thus did not breach the terms, either while it had accounts or after it closed them. The court also refused to read Meta's survival clause as banning former users from ever scraping public data.

In X Corp. v. Bright Data, Judge William Alsup dismissed X's complaint in May 2024 on a different ground. X does not own its users' posts; its terms take only a non-exclusive licence. Using contract and tort law to stop others copying those posts, the court held, conflicted with the federal Copyright Act, so the claims based on scraping and selling of data were preempted. In the court's words, " X Corp. wants it both ways: to keep its safe harbors yet exercise a copyright owner's right to exclude". This is called copyright preemption: federal copyright law displaces state-law claims that protect the same rights.

hiQ lost on contract

The hiQ litigation shows the other side. After the Ninth Circuit's CFAA ruling, the case went back to the trial court on LinkedIn's counterclaims. The court held at summary judgment that LinkedIn's User Agreement unambiguously and enforceably prohibits unauthorized scraping and creation of fake accounts, and hiQ had both used fake accounts and agreed to the terms. In December 2022 the parties filed a stipulated judgment: $500,000 against hiQ and a permanent injunction requiring hiQ to permanently delete any and all LinkedIn member profile data and the code built to use it. The scraper that won the famous ruling lost its data.

Reddit v. Anthropic (2025 to 2026)

Reddit sued Anthropic in California state court in June 2025, alleging Anthropic scraped Reddit despite Reddit's user agreement and its requests to stop ( filing). Reddit pleaded five causes of action, none of them copyright: breach of contract, unjust enrichment, trespass to chattels, tortious interference with contract and unfair competition. Anthropic moved the case to federal court, arguing, as Bright Data had, that the claims were really copyright claims. In March 2026 the federal court disagreed and sent the case back, holding that each claim had an "extra element" beyond copying and that " Courts have repeatedly held that violations of website terms of use are not preempted by the Copyright Act." In September 2026 the state court allowed three of Reddit's five claims to proceed.

So the same federal court in San Francisco has gone both ways on preemption: X's claims were preempted, Reddit's were not. The difference the courts drew is between a contract with real mutual obligations, which survives, and a claim that is only "you copied our content", which copyright law displaces. A scraper relying on preemption is relying on an open question.

4. Trespass to chattels

Trespass to chattels is an old tort: interfering with someone else's personal property. Courts have applied it to bots since the early days of the commercial web, and site operators still plead it against scrapers (Reddit did against Anthropic). But California, where most of these cases are filed, requires real harm. Quoting the state's Supreme Court in Intel v. Hamidi, the court in X Corp. v. Bright Data held that " intermeddling is actionable only if the chattel is impaired as to its condition, quality, or value" or the possessor loses its use for a substantial time, and dismissed X's trespass claim because the complaint only recited those words.

The practical effect: trespass reaches scraping that burdens the servers, such as high-volume crawling that slows a site or raises its costs, and that the site can prove. A polite crawler reading public pages at a modest rate rarely meets the standard. A crawler that disguises itself to evade blocks and hits a site hard is a better target, which is part of what publishers allege against some AI crawlers today (see section 7).

5. Copyright and fair use

Scraping text, images, video or code copies it, and most of it is protected by copyright the moment it is written. A scraper that trains on copyrighted material needs either a licence or a defence, and the defence is fair use. Whether training qualifies is the subject of our separate guide, Fair Use and AI Training. In short: two federal judges held in 2025 that training generative models on books was fair use, the Third Circuit held in 2026 that copying a competitor's editorial content to train a rival tool was not, and every court so far has treated how the copies were obtained as a separate question. Copies taken from pirate sources were held inherently, irredeemably infringing in the Anthropic case, which then settled for $1.5 billion plus interest.

For scraping specifically, two points follow. First, the risk depends on what you scrape and what you do with it. Scraping facts (prices, addresses, specifications) copies little protected expression. Scraping articles to build a product that answers readers' questions with them invites the strongest claims, because it substitutes for the source. That is the theory behind the wave of suits by publishers against answer engines: Perplexity faces suits by Dow Jones and the New York Post (where the court denied Perplexity's motion to dismiss in full), Britannica and Merriam-Webster (who allege that "PerplexityBot" crawls and scrapes their sites), the New York Times and CNN. Wikihow sued OpenAI in 2026 for scraping its content "whether directly or through intermediary datasets".

Second, copyright cuts both ways for site operators. A platform that does not own its users' posts cannot sue for infringing them, and, as X Corp. v. Bright Data showed, may find its state-law claims preempted because they look like copyright claims. The parties with the strongest copyright position against scrapers are those who own the content outright: publishers, reference works, and companies whose own staff wrote the material.

6. Anti-circumvention: getting around technical barriers

What it forbids

The Digital Millennium Copyright Act of 1998 added a separate prohibition to copyright law: you may not circumvent a technological measure that controls access to a copyrighted work, and you may not sell tools or services that do. A measure "effectively controls access" when in the ordinary course of its operation, it requires the application of information, or a process or a treatment, with the authority of the copyright owner, to gain access to the work. Circumvention is a violation in itself, separate from any copying, so a scraper can face it even if its later use of the content would be fair. That makes this provision, DMCA section 1201, the most dangerous theory for a scraper that gets past technical barriers.

Robots.txt is not a lock

In December 2025, the judge handling the consolidated OpenAI cases in New York refused to let Ziff Davis add a claim that OpenAI circumvented its robots.txt files. A robots.txt file only works if the crawler chooses to read and obey it: " If the bot creator does not take this affirmative step of choosing to heed the requests embodied in Ziff Davis's robots.txt files, the robots.txt files have no effect on the bot's ability to access Ziff Davis's works." A request is not an access control, so ignoring it is not circumvention.

CAPTCHAs and bot challenges can be

The opposite result came in July 2026. Reddit sued Perplexity and three scraping services, SerpApi, Oxylabs and AWMProxy, alleging they obtained Reddit content by getting around anti-bot measures, including Google's "SearchGuard", which sends a JavaScript challenge or CAPTCHA to automated queries before showing search results that contain Reddit snippets ( complaint). On 31 July 2026, Judge Paul Engelmayer predominantly denied the motions to dismiss: he sustained circumvention claims against both Perplexity and SerpApi, a claim against SerpApi for offering circumvention services, and a civil conspiracy claim against both, while dismissing Reddit's unjust enrichment and unfair competition claims as preempted by copyright. The press summary is that Reddit plausibly pleaded that Perplexity conspired with at least one of the scrapers to bypass access controls.

The law is not settled even here. Google brought its own circumvention suit against SerpApi in California, and that court dismissed Google's claims without prejudice in July 2026, meaning Google may try again. But the direction for a scraper is clear enough: a scraper that solves, spoofs or routes around challenges built to stop bots is in different territory from one that reads open pages, and paying a service that does it can make you a co-conspirator.

7. Robots.txt: not law, but evidence of notice

What it is

Robots.txt is a plain text file at the root of a website, following the Robots Exclusion Protocol, that tells crawlers which parts of the site they may visit. AI companies publish the names of their crawlers (GPTBot for OpenAI, ClaudeBot for Anthropic, Google-Extended as Google's switch for training Gemini, CCBot for Common Crawl), so a site can shut out training crawlers while staying open to search engines. No US statute makes a robots.txt line binding, and the Ziff Davis ruling held it is not an access control.

What it does in law

Robots.txt still matters, as evidence. It records that the site asked crawlers to stay away, when, and which ones. That bears on several of the theories above: whether a scraper had notice of the site's wishes (relevant to contract formation and to trespass), whether infringement was willful (which raises statutory damages to up to $150,000 per work), and whether the scraper acted in bad faith (a consideration some courts still weigh under fair use). Disguising a crawler to get around robots.txt is itself the kind of fact plaintiffs now plead: News Corp alleges that Brave disguises its web crawlers to evade publisher blocks, and Ziff Davis alleged that OpenAI reconfigured GPTBot to bypass robots.txt after finding compliance hindered its work (an allegation the court took as true for its ruling, not a finding). Reddit's own robots.txt, as the court described Reddit's pleading in July 2026, prohibits unapproved web crawlers from accessing any part of the site.

How many sites say no

Our AI crawler census reads the robots.txt of the most visited sites on the open web, 4,000 a morning. In its current pass it has read 6,094 robots.txt files. Of them, 11.4% shut at least one AI crawler out of the whole site, and 8.4% shut out two or more, at least one of them a training crawler.

The census measures the most visited sites, which are not a random sample of the web, and a site that blocks one crawler often blocks several. Academic work points the same way for the data AI is trained on: the "Consent in Crisis" study found that robots.txt restrictions cut off about 5% or more of all tokens in C4, and 28% or more of its most actively maintained sources, a widely used training corpus, between 2023 and 2024. The daily slices are on record too, for example the 17 October 2026 slice.

For a scraper, a disallow line is a warning that the site will treat crawling as unwanted and may say so in court. For a holder, it is cheap evidence worth having, but not a wall.

8. Privacy law: Clearview and Illinois BIPA

Public is not the same as free to collect

Privacy law asks a different question from all the theories above: not whether the scraper was allowed onto the page, but whether it may collect and use information about people. Several US laws apply regardless of whether the data was public. The sharpest is Illinois's Biometric Information Privacy Act (BIPA), which requires notice and written consent before a private company collects a biometric identifier (a face geometry, a voiceprint, a fingerprint), and which flatly provides that no private entity may sell, lease, trade, or otherwise profit from a person's biometric identifier or information. BIPA gives individuals their own right to sue, with statutory damages per violation, which plaintiffs plead at $1,000 per negligent violation or $5,000 per intentional or reckless one.

The Clearview case

Clearview AI is the cautionary tale. As the federal court in Chicago described it, the company assembled a massive biometric database of individuals by "scraping" their photos from publicly available websites and collecting their biometric facial geometry, then sold searches to law enforcement and retailers. The photos were public; the face geometry was not something anyone had consented to give. The resulting nationwide class action, consolidated in Illinois, settled with the class receiving a payout from a 23% equity stake in Clearview, approved in March 2025, because the company said it could not afford a cash payment large enough. Regulators abroad fined it as well: the Dutch data protection authority imposed EUR 30.5 million, and a UK tribunal held that UK data protection law does reach Clearview, keeping alive the regulator's £7.5 million fine, which Clearview is still challenging.

BIPA suits over AI training have not stopped. In October 2026 two proposed class actions in Cook County accused delivery-robot companies of scanning customers' faces and hands and retaining the data for artificial intelligence training.

Beyond biometrics

State comprehensive privacy laws (California's among them) treat some "publicly available" information more leniently, but the definitions are narrow and differ by state, and they do not cover everything a scraper sweeps up. Children's data, health data and data about EU residents carry their own rules. Our guide on whether it is legal to sell customer data covers the privacy layers in detail. For scraping, the rule of thumb is that faces, voices and anything identifying a person need a consent basis that a scraper almost never has. Our pages on images and audio and speech show how these kinds of data are bought and sold.

9. The leading cases at a glance

10. Why scraped data is hard to resell

You cannot license what you never had

The cases above are mostly about whether scraping exposes the scraper. A different question decides whether scraped data has any value in a sale: what rights does the scraper have to pass on? Usually none. Scraping does not transfer copyright from the authors; it does not transfer the platform's licence from its users; and it does not create consent from the people in the data. A scraper that wins every lawsuit still holds only copies, plus whatever defence it might raise if sued. It cannot grant a buyer a licence to rights it does not hold, and a defence such as fair use belongs to each user and each use: it does not travel with the files.

This is the core of the licensing market. Holders who own their data, or hold clear licences to it, can grant rights; scrapers can only sell copies. Reddit's own terms make the point from the platform side: even partners with access to its data API may not use the content for AI training unless they have entered a separate agreement, which is what its paying partners sign. Its deals with Google and OpenAI are each reported at about $60 million per year; our guide to Reddit's AI data licensing deals covers them.

What buyers now require

Buyers know this, and their published terms show it. Data buyers ask holders to confirm, in writing, that they have the right to license what they deliver:

Licence agreements typically add a warranty "That the data was collected in compliance with applicable law", backed by an indemnity for third-party claims. A scraper cannot give those promises honestly, and a buyer that accepts scraped data takes on every claim the scraper faced. California now also requires developers of public generative AI systems to post a summary of their training data that includes the sources or owners of the datasets and whether they were purchased or licensed, so where data came from is becoming a public fact about a model.

The remedies follow the data

Courts can order scraped data destroyed. hiQ had to delete the LinkedIn profiles it scraped and the code built to use them ( judgment). A buyer whose model or product rests on such data inherits that risk, which is why sophisticated buyers increasingly treat provenance as part of the price. Buyers have even been offered stolen archives: one AI data executive described being offered a large company's codebase that turned out to be an illicit copy shopped to every data vendor.

The price gap

The market already prices the difference. Scraped datasets are sold off the shelf for fractions of a cent a record (one large web data vendor lists a Wikipedia dataset at about $0.0025 per record), while archives licensed by their holders command the deal sizes on our deal ledger; the price board shows both ends. In robotics data, footage with a known source trades at several times the price of data of unknown origin. Of the vendors on our vendors register, 17 say they build from data licensed from its owners.

Worked example: the scraper and the source

This is an illustrative example, not a real company. A two-person start-up has scraped several years of public listings and reviews from a niche marketplace, logged out, at a modest crawl rate, and wants to sell the dataset to an AI company. The marketplace itself, a 60-person company, holds the same listings plus the private data behind them: messages between its users, transaction outcomes, dispute records.

The scraper's position. Under the CFAA it is probably safe (public pages, no login). Under contract it is probably safe if it never opened an account (Meta v. Bright Data). It has no trespass problem at a modest crawl rate. But the reviews are copyrighted by the people who wrote them, the marketplace's robots.txt disallowed its crawler, and it has nothing to show a buyer except files. It cannot warrant the right to license, cannot warrant consent, and cannot indemnify against claims by the marketplace or its users. A careful buyer will either refuse or pay a fraction of what the data would be worth with rights attached.

The marketplace's position. It can license the public listings and, after de-identification, the private records a scraper never saw, with a clear rights file and a warranty it can actually give. It can also put the scraper's buyers on notice. For a buyer, the marketplace's data is worth more not because it is bigger but because it comes with rights the buyer can rely on. Our guide to selling company data to AI labs covers how holders do this.

Questions people ask

Is web scraping legal in the US? Scraping publicly accessible pages is generally not a federal crime after Van Buren and hiQ. It can still breach a contract you accepted, infringe copyright, violate anti-circumvention law if you bypass technical barriers, or breach privacy laws. Whether a given scrape is lawful depends on how you reach the data, what it is and what you do with it.

Is scraping public data a crime under the CFAA? Usually not. The Ninth Circuit held that a public website has no gates to lift or lower, so scraping its public pages is unlikely to be access "without authorization". Scraping behind a login you are not entitled to, or with fake accounts, can still be.

Can a website's terms of service stop scraping? Yes, against people who agreed to them. Courts enforce terms a user clicked to accept, and hiQ was held bound by LinkedIn's User Agreement. Terms behind a footer link rarely bind a logged-out bot, and Meta lost its contract claims against Bright Data for that reason.

Is ignoring robots.txt illegal? Not in itself. A New York federal court held that robots.txt is not an access control under anti-circumvention law. It is evidence that the site objected, which can support claims of willful infringement or bad faith, and disguising a crawler to evade it is now pleaded in lawsuits.

Can I sell data I scraped? You can sell copies, but you cannot license rights you do not hold: the copyright stays with the authors and consent stays with the people in the data. Buyers now ask holders to warrant the legal right to license the data for AI training, which a scraper usually cannot honestly give.

Did LinkedIn win against hiQ? In the end, yes. hiQ won the CFAA argument in the Ninth Circuit, but the trial court held LinkedIn's terms enforceably banned its scraping, and the case ended with a stipulated $500,000 judgment and an order to delete the scraped data.

Is it legal to scrape photos of people's faces for AI? It is very risky. Illinois's biometric privacy law requires consent before collecting face geometry and bans profiting from biometric identifiers, and scraping public photos led to the Clearview settlement and fines abroad. Public availability is not a defence under these laws.

What is the Reddit v. Perplexity case about? Reddit alleges that Perplexity and three scraping services got around anti-bot measures to obtain Reddit content through Google search results. In July 2026 the court let the circumvention and conspiracy claims proceed while dismissing claims it found preempted by copyright.

What this means for you

If you are thinking of scraping for AI, stay on the public side of every login, do not accept terms you intend to break, do not solve or route around bot challenges (or pay anyone who does), keep crawl rates modest, honour robots.txt as a record of what sites want, and keep faces, voices and other personal data out entirely. Even then, assume you are acquiring copies, not rights, and that a buyer will discount them accordingly.

If you buy data, ask every vendor how each dataset was obtained, and prefer data that comes with a licence from whoever owns it. The buyer board and the vendors register show who sells licensed data, and our guide to training data provenance covers what to ask for.

If you hold data that others scrape, put a clickwrap no-scraping clause in front of anything valuable, use real technical barriers rather than requests alone, keep a robots.txt that names AI crawlers, and consider whether licensing on your own terms beats litigating. Your private records, the part no crawler can reach, are often the most valuable thing you hold. The valuation gives a first estimate with no data upload, the readiness check shows what a buyer will ask about your rights, and the licence terms register and our guide to data licensing agreements explain the clauses that protect you once you do license.

AIData.Broker acts as the holder's agent: the holder itself licenses or sells its data to the buyer, and we do not take ownership of it. We run sales only for holders based in the United States. More on how buyers source data is in how AI companies get training data and on the rights hub.

This guide is research from public sources, not legal advice.