As AI bots take over the internet and displace human eyeballs, one of the biggest web-infrastructure companies has an idea about how to charge them. Cloudflare recently rolled out what is essentially a bot paywall that would allow its customers to charge AI models for scraping its customers’ content. Online retailers, travel companies, and publishers, which spent decades wrangling with Google over how to get paid for their websites, now face the same challenge as LLMs scrape their content.
The internet is moving toward a point where “every interaction, every transaction, is a chance for commerce,” Cloudflare’s chief strategy officer, Stephanie Cohen, told Semafor, adding that more than half of its web requests now come from agents. Cloudflare’s new architecture allows content owners to set a minimum price and AI crawlers to set a maximum bid, and lets the magic of a two-sided marketplace take it from there.
That doesn’t mean the subscription businesses that media companies labored to build will collapse, Cohen said. “I don’t think everyone wants to read the average,” she said, which is good news for scoopy and voicey publications (hi.)
