How Do AI Companies Get Their Training Data?

AI companies get their training data through six channels: they crawl the open web, license content from publishers and platforms, pay experts and contractors to make data to order, buy or license records that companies already hold, build simulated environments and synthetic data, and use what their own users write and upload. The first channel built today's models and is still the largest by volume: one monthly Common Crawl archive holds 2.17 billion web pages. It is also the one closing fastest, with 11.4% of the most visited sites in our census shutting at least one AI crawler out. The money has moved to the other five: the large data vendors' newest annual revenues and run rates add up to $8.1B, Reddit signed $203.0 million of data licences in a single month, and Google bid $10 million for one bankrupt airline's emails and chats.

This guide takes each channel in turn: how it works, how big it is, what it costs, what legal exposure it carries and which way it is moving. It is written for the founder, CFO or counsel of a company that holds data and wants to understand where its data fits, and for anyone watching the market who wants the mechanics rather than the headlines. Our supply map shows where data sits, kind by kind; this guide shows how it moves, channel by channel. It closes with how the mix is shifting and what that means for a holder.

AIData.Broker acts as the holder's agent: the holder itself licenses or sells its data to the buyer, and we do not take ownership of it. This guide is research from public sources, not legal, tax or investment advice.

Contents

1. The six channels at a glance
2. Crawling the open web
3. Licensing content from publishers and platforms
4. Commissioning human and expert data
5. Buying company records and archives
6. Building environments and synthetic data
7. Their own users' data
8. How the mix is shifting
9. Questions people ask
10. What this means for you

The six channels at a glance

A large language model learns in stages. Pretraining feeds it trillions of words so it learns language and facts; Alibaba's Qwen3 models, for example, were pretrained on approximately 36 trillion tokens (a token is a word or a piece of one). Post-training then teaches it to follow instructions, reason and do tasks, using far less data, but data that is much harder to get: worked answers written by experts, judgments about which answer is better, and practice in simulated software. Each stage draws on different channels, and the channels differ in what they cost and who gets paid.

The table summarises the six channels before the sections take each apart. "Who gets paid" is the column that matters most to a holder: only three channels pay the people or companies that hold the data, and they are the three growing fastest.

1. Crawling the open web

How crawling works

A crawler is a program that downloads web pages by following links, the same way a search engine indexes the web. Two kinds feed AI. The first is Common Crawl, a nonprofit archive that has crawled the web for years and publishes each crawl for anyone to use; its September 2026 archive holds 2.17 billion web pages, or 361.4 TiB of uncompressed content, from 40.4 million hosts. Most large models start from cleaned versions of archives like it. The second kind is each lab's own crawler: OpenAI's GPTBot, Anthropic's ClaudeBot, Meta's meta-externalagent, ByteDance's Bytespider, plus the switches Google and Apple offer (Google-Extended, Applebot-Extended) for refusing AI training specifically.

Crawling is cheap to collect and expensive to clean, and it is running out. Epoch AI estimated the stock of quality public human text at around 300 trillion tokens and projected that models would fully use it between 2026 and 2032. Of the 2.17 billion pages in the September archive, only 587.2 million URLs had never been crawled before: most of each new crawl is the web the models have already read.

How many sites now block AI crawlers

A site tells crawlers what they may fetch in a public file called robots.txt. Our AI crawler census reads that file for the 300,000 most visited sites, 4,000 a morning, and records which AI crawlers each one shuts out of the whole site. In the current pass it has read 6,094 sites, and 11.4% block at least one AI crawler; 8.4% block two or more, one of them a training crawler. Most of the web still lets the crawlers in. The refusals matter more than their share suggests, because they are concentrated among the sources with the most valuable text.

The blocking is concentrated where it hurts. A 2024 academic audit of the C4 dataset (one of the standard cleaned web corpora) found that robots.txt restrictions added in a single year cut off about 5% or more of all its tokens, and 28% or more of its most actively maintained sources, while 45% of C4 was restricted by terms of service. Publishers' reason is arithmetic: Cloudflare measured Anthropic's crawlers fetching as much as 500,000 pages for every visitor they referred back in 2025, against a peak of 3,700 to one for OpenAI. A crawler that takes and sends nothing back is a cost.

What it costs, and the legal exposure

Crawling carries no licence fee, so its cost is legal risk, and that risk is now priced. The clearest price so far came from books rather than web pages: in Bartz v. Anthropic, the court held that training on lawfully acquired books was fair use but sent the pirated library copies to trial, and the case settled for $1.5 billion, about $3,000 per work. In Kadrey v. Meta the court granted partial summary judgment for Meta on fair use. In Thomson Reuters v. Ross, a non-generative legal search tool, the Third Circuit affirmed for Thomson Reuters against fair use. The New York Times' case against OpenAI and Microsoft sits at cross-motions for summary judgment, and the US Justice Department filed a statement urging that training on written works is fair use, which our record notes is non-binding. We track 83 lawsuits, rulings and settlements on the cases page; the fair use guide explains how the courts are splitting.

Robots.txt itself is a request, not a lock. In Ziff Davis v. OpenAI, the court dismissed a claim that ignoring robots.txt breached the anti-circumvention rules, on the reasoning that "robots.txt directives are merely requests". Platforms have turned to contract and scraping law instead: Reddit sued Perplexity and three scraping firms, and a judge largely rejected their attempt to dismiss; News Corp sued Brave alleging it disguises its crawlers to evade publisher blocks. The guide to whether scraping data for AI is legal covers the law in detail.

Direction

Crawling will remain the base layer, because nothing else supplies trillions of tokens. But the share of valuable text a crawler can take without asking is shrinking, the sites that matter most are the ones refusing, and every major lab now faces at least one suit over what it took. The labs' answer has been to pay for the content they most need, which is the second channel.

2. Licensing content from publishers and platforms

When a publisher blocks a crawler or sues, the lab's alternative is a licence: a contract that pays for access to an archive, a live feed, or both. Our deals tracker records 111 disclosed licensing deals, 56 of them for training models; another 40 license content to be shown in AI answers rather than to train on, and only 19 disclose a dollar amount. OpenAI alone has signed 29.

These deals pay the large holders of text, images and video: news publishers, academic publishers, stock-image libraries and user-generated platforms. They rarely pay the people who wrote the content directly, and when they do, the share depends on how the licence was set up: where authors had to opt in, HarperCollins paid them $2,500 a title, while opt-out arrangements paid authors far less.

The largest disclosed deals

Sources for the deal structures below: recurring access, minimum plus variable, paid up front, one-time deals.

How the deals are structured

Three patterns recur. Most large licences grant recurring access for two to three years rather than a one-off copy: Reddit's January 2024 licences ran two to three years with at least $66.4 million recognised in the first year. Many pair a guaranteed minimum with a variable part: OpenAI pays Dotdash Meredith a fixed amount with a variable component on top, and Adweek reported that most of OpenAI's publisher deals use fixed and variable components. And a large share of the money arrives up front: Getty recognised about 62 percent of two multiyear deals in the quarter they were signed, then guided the next year's revenue lower, attributing the decline to one-time revenue from those licensing deals.

The licensor side has become a real business line for some. Shutterstock's data, distribution and services revenue, which includes licences for AI training, reached $203.3 million in 2025. Wiley reports lifetime AI revenue over $110 million, and its research business sells AI licences that include content Wiley licensed from other publishers, paying them royalties: a smaller publisher can reach the market through a larger one.

Legal exposure and direction

For the lab, a licence replaces copyright risk with contract terms, which is the point of signing. For the licensor, the exposure is whether it holds the rights it licenses: an academic press licensing authors' books, or a platform licensing its users' posts, has to rely on its own terms with those authors and users.

The direction is toward more deals, smaller and more specific, and toward live feeds for display as much as archives for training. Google's News AI pilot pays participating publications for the delivery and use of enhanced content; Meta signed a run of news deals with terms not publicly disclosed. For the history and numbers of the two largest groups, read the guides to Reddit's AI data licensing deals and news publishers' AI licensing deals.

3. Commissioning human and expert data

Post-training needs data that does not exist anywhere until someone makes it: a doctor's worked answer to a hard diagnostic question, a lawyer's ranking of two contract summaries, an engineer's solution to a coding task with tests that check it. Labs buy this through human data vendors, companies that recruit experts and contractors, design the tasks, check the work and deliver it. The technique most associated with it is RLHF (reinforcement learning from human feedback): people judge which of a model's answers is better, and the model is trained toward the preferred ones. Today the work runs well beyond ranking, into writing full tasks, red-teaming (trying to make a model misbehave) and grading long agent runs.

This is the paid channel with the most revenue on the record, and the one where the money is best documented. The large vendors' newest annual revenues and run rates add up to $8.1B, and the experts themselves are paid well by the hour. Our expert work page and rates page carry the detail; the summary below shows the shape.

Scale and price

The workforce is large. Handshake was paying ~$60M a month to over 30,000 experts by late 2025; Mercor shows $4M+ in daily payouts on its homepage and refers to nearly five million experts on its platform; Scale's Outlier says it has 900K+ AI trainers. Across advertised roles, experts earn $55 to $101 per hour of work, and Mercor states an average contracted rate of $112/hr.

The vendors keep a large share of what labs pay. For one expert hour, a lab pays Mercor $160 to $187 an hour, of which Mercor keeps $48 to $75 an hour, or 30% to 40% of each dollar. Appen, which reports its margin after paying its crowd, kept 40.3 percent in 2025. Prices per task are rising as tasks get harder: Mercor's chief executive described tasks priced from $50 to $10,000, and coding tasks across several vendors pay contributors the equivalent of about $45 to $100 an hour.

Legal exposure and direction

The legal exposure in this channel mostly sits with the vendor, which recruits the experts, contracts with them and holds a great deal of personal data about them. Mercor's own update on a security incident said that of its nearly five million experts, only a limited subset had sensitive information affected. For the lab, the risk is concentration: one customer accounted for about 58 percent of Innodata's 2025 revenue, which shows how few buyers set the price.

The direction is toward longer, more expert tasks and toward environments (channel 5), which the same vendors now build. It is also toward the records companies already hold (channel 4): several of these vendors, having paid experts to recreate how work is done, have started buying the real thing. The guide to AI training data companies maps who does what.

4. Buying company records and archives

The newest channel is the one this site was built around. A company's internal records (its email, chat, tickets, code with its review history, CRM and documents) show how real work gets done over years: who asked what, what went wrong, how it was fixed. No crawler can reach them and no expert can recreate them at scale. Since 2025, labs and the vendors that supply them have started paying companies for these records, licensing them under contract and de-identifying them before use.

The channel is small next to the vendors' revenue, but it is where much of the newest money is going. Our buyer board lists 24 buyers that publish a way in for company records. micro1 committed to spend $1 billion over the next 12 months acquiring and licensing them, and told a bankruptcy court it had closed more than 50 data transactions in 45 days. SimpleClosure, which helps startups shut down, counts 9 times more AI buyers than in 2025.

What the records fetch

The best-documented sale is Spirit Airlines' archive, sold in its bankruptcy. The archive held about 100 million emails and 500,000,000 Teams items. Google opened at $5 million, three qualified bidders took it to $10.0M in one session, Mercor was named the $7.5 million alternate, and micro1 later filed $12.5 million. At the small end, closed startups sell their workspaces for far less: SimpleClosure's early deals ran $10K to $100K per company, and one founder netted roughly $25,000 for a codebase. SimpleClosure now reports over 150 deals, with 47% of its deals including code or data sales. Turing's chief executive said it bought five to ten failed-startup codebases at an average in the tens of thousands of dollars.

Between those ends sit the published programs, with tiers such as micro1's $100k+, $500k+ and $1M+ and ranges such as Handshake's $100K to $4M+. The comparison of company data programs sets their terms side by side, and the guide to what company data is worth explains how they price.

Legal exposure

This channel carries the heaviest legal exposure for the holder, because company records are full of other people: customers, employees and suppliers. In the Spirit sale the parties agreed that passenger transaction, refund and inflight purchase systems would not be transferred in any form, and the sale hearing was adjourned repeatedly while the estate negotiated with objectors over employee information. Five third parties objected that their own data sat inside the archive. When Character.AI filed for bankruptcy in October 2026, its estate held more than 21.6 million US user accounts, and what could happen to those chats became part of the coverage. The rights guides cover each risk: customer data, employee emails and chats and sales in bankruptcy.

Direction

Up, and broadening from closed companies to operating ones. The supply is large and mostly untouched: our closures register records 149 technology companies closing, sold or gone dark, and every operating company with a few years of records is a potential holder. See company records for what buyers want and estates for companies winding down.

5. Building environments and synthetic data

A reinforcement learning environment (an RL environment) is a simulated piece of software, such as a copy of a website, an email client or a project tracker, with tasks and an automatic way to check whether the model completed them. The model practises by trial and error, over and over, and is rewarded when it succeeds. This is how labs are teaching models to act as agents that use software rather than just talk. Synthetic data is data generated by models themselves, often seeded and checked with real examples; environments and synthetic data overlap, because an environment generates practice data every time a model uses it.

Labs build some environments in-house and buy many. Epoch AI's survey of the business found contracts often six to seven figures per quarter, with $300k to $500k seen by one researcher; a replica of a website costs about $20K, and about $300K for a Slack-grade clone; individual tasks run $200 to $2,000, with $20k per task rare but possible. SemiAnalysis reported that OpenAI purchased hundreds of sites of this kind.

Who builds them, and what they cost to make

Our vendor register lists 89 of its 150 vendors as selling RL environments. They range from Halluminate, which raised a Series A with four of the top five closed-source US labs as customers by Fortune's report, to open bounty programs: Prime Intellect crowdsourced 400+ environments for bounties of $100 to $500, and about $1,000 to $5,000+ for application-only tasks. Mercor bought Deeptune, an environment builder, in July 2026.

The margin between making and selling is wide. Mercor's chief executive gave an example $1,500 cost basis per task (ten hours of expert time) and described labs paying $2,000 per task to order legal tasks at scale, and about $500K in compute per post-training run on 1,800 tasks. Across the prices we record, buyers pay 9.5x what the people who make datasets, tasks and environments are paid, per task. Exclusivity is priced explicitly: two founders told Epoch that exclusive deals cost roughly 4 to 5 times non-exclusive ones.

Legal exposure and direction

Environments carry the least legal exposure of any channel for the lab, because the lab or its vendor builds them and owns what it builds. The exposure moves to what the environment is built from. A replica of real software can be built from scratch, but a growing number of environments are built from real company records: several buyers on our board, such as Ooak, Collinear and The General Data Company, turn a company's anonymised records into a simulated copy of that company for agents to work in. That makes this channel the main new demand behind channel 4.

The direction is fast growth with fast decay: one investor's analysis put an environment's useful life at a half-life of about six months, and found the same customer rarely buys the same environment twice. That churn is why environment builders keep needing fresh material. The guide to RL environments goes deeper, and datasets and environments shows what sells.

6. Their own users' data

The last channel is the one closest to home: what people write, post and upload on the AI company's own products, or on the platforms the AI company owns. A social network's posts, a professional network's profiles, a chatbot's conversations and a photo app's uploads are all training data its owner already holds. Using them needs no licence fee and no crawler, only a change in the terms users agreed to.

That change is where the friction is. In the EU, under its data protection law, the Irish Data Protection Commission has engaged with each large platform in turn. Meta paused its plans to train on adults' public Facebook and Instagram posts in June 2024, then resumed from 27 May 2025 after changes, under monitoring. LinkedIn told the regulator in March 2025 that it would train its own generative AI models on EU members' personal data from early November 2025. X agreed to suspend processing EU users' public posts to train Grok, and the regulator opened an inquiry into whether that processing was lawful. It opened another into Google's PaLM 2 training.

The US rule: old promises bind new uses

In the US, the main federal check is the Federal Trade Commission's power over unfair and deceptive practices, and in February 2024 the FTC warned the AI industry directly. Companies, it wrote, may try to rewrite their privacy policies to free up user data for AI, and "any firm that reneges on its user privacy commitments risks running afoul of the law"; adopting more permissive practices through a "surreptitious, retroactive amendment" to terms of service may be unfair or deceptive. The practical rule follows: data is governed by the policies in force when it was collected, unless people give fresh consent or the data is fully anonymised.

That rule applies to every holder, not just to AI companies. A business that wants to license its customers' data for training is bound by what its own privacy policy said when it collected that data, which is why our rights checklist asks what your privacy policy said at the time before anything else. Consumer groups are testing the edges in court too, in suits such as the Dutch Consumentenbond action over Meta's AI glasses.

Direction

Expanding, but with opt-outs, regulators and a written record of every change. The large consumer platforms have the most user data and the most to lose from getting it wrong, so this channel grows through announced policy changes, not quietly. For anyone else, it is a reminder that the data a company collected under one promise cannot be repurposed under another without care.

How the mix is shifting

Put the six channels side by side and one movement stands out: from data taken to data paid for. The open web is not disappearing, but its best sources are refusing crawlers, its stock of quality text is being used up, and taking it without asking now carries settlements counted in billions. At the same time the work labs most want done (reasoning, coding, acting as agents in real software) needs data that is private, expert or simulated. Former OpenAI researcher Will DePue forecast, as an opinion rather than a budget, that leading labs could spend more than $100 billion a year on data by 2030.

What that means for a holder

For a company that holds data, the shift has three consequences. First, data behind a login has become the scarce input: what a crawler could take for free is worth little, while records no crawler can reach are what three of the six channels now pay for. Second, how records connect matters more than how many there are: buyers name how traceable the records are, and how well they connect across platforms as what raises a price, and one estimator values the same company at $85k to $121k with six connected tools but $16k to $22k with three. Third, rights decide what can be sold at all: the same privacy promises, employee messages and vendor terms that slowed the Spirit sale apply to every company's archive.

It also means a holder has more than one kind of buyer. A publisher's archive may suit a lab's licensing team; a software company's code and tickets may suit an environment builder; a services firm's CRM and email may suit a company-records program. The what labs buy hub shows which kinds of data each kind of buyer names.

Questions people ask

Where do AI companies get their training data?

From six channels: crawling the open web, licensing content from publishers and platforms, paying experts to make data to order, buying records companies already hold, building simulated environments and synthetic data, and using their own users' posts and uploads. The open web is still the largest by volume, with 2.17 billion pages in one monthly Common Crawl archive, but most new spending goes to the paid channels.

Do AI companies pay for training data?

Increasingly, yes. Reddit disclosed $203.0 million of licences signed in one month, the large data vendors' revenues and run rates add up to $8.1B, and 24 buyers publish programs that pay companies for their records. What they still do not pay for, in most cases, is the public web they crawl.

Is it legal for AI companies to scrape websites?

It depends on what is taken, how and where, and US courts are split. Training on lawfully acquired books was held fair use in Bartz v. Anthropic, but pirated copies went to trial and the case settled for $1.5 billion; the Third Circuit ruled against fair use in a non-generative case. The scraping guide covers contract and privacy law as well.

Can I stop AI companies from crawling my website?

You can ask, through robots.txt: 11.4% of top sites shut at least one AI crawler out. But a court has held that robots.txt directives are merely requests, not access controls, so enforcement rests on terms of service and scraping law. Our crawler census shows which crawlers sites block.

Do AI companies train on users' posts and chats?

Many platforms do, after changing their terms: LinkedIn began training on EU members' data in November 2025 and Meta resumed training on EU users' public posts in May 2025, both under regulators' monitoring. In the US, the FTC has warned that changing terms retroactively and surreptitiously to allow such use may be unfair or deceptive.

How much do AI companies pay for data?

It ranges over many orders of magnitude, by channel. Experts earn $55 to $101 per hour of work; environment tasks sell for $200 to $2,000; a closed startup's workspace has fetched $10K to $100K; a large company's archive drew bids up to $12.5M; and the largest reported content licences run to about $60 million a year. The prices guide and price board carry every published figure.

What is an RL environment?

A simulated piece of software (a website, an inbox, a ticket tracker) with tasks and an automatic check, in which a model practises until it succeeds. Labs buy them in contracts often six to seven figures per quarter, and the most realistic are built from real company records. See the guide to RL environments.

What this means for you

If you hold data, the question is no longer whether AI companies want it but through which channel, on what terms and at what price. The steps below apply whatever kind of data you hold.

1. Find your channel. Public content you publish is exposed to crawling; decide whether to block, license or both, using the crawler census and the licensing deals tracker. Private records belong to channels 2, 4 and 5: see what labs buy and the supply map.
2. Know who buys it. The buyer board lists every buyer with a published way in, and the vendor register shows which vendors say they pay for data; the guide to AI training data companies explains what each kind does.
3. Get a first reading of value. Value your data against published prices before any buyer names a number.
4. Check the promises you made. Read the privacy policy, customer contracts and employee terms in force when the data was collected; the readiness check works through the rights questions buyers' counsel ask.
5. Compare before you choose. Every channel that pays holders rewards a holder that can show the same description to several buyers; the comparison of company data programs explains why, and the guide to selling company data walks through the process.

We run that process for US-based holders that want an agent on their side; how we work and what it costs are on the pricing page.