CloudFlare, a cloud infrastructure provider that serves 20% of the web, announced on Tuesday the launch of a new market that rethinks the relationship between website owners and AI companies.
Last year, CloudFlare launched a publisher-friendly tool to deal with the ramping rise of AI crawlers, including a one-click solution that blocks all AI bots and a dashboard that shows how AI crawlers visit their site. In an interview in 2024, CloudFlare CEO Matthew Prince told TechCrunch that these products lay the foundation for a new type of market where publishers can distribute content to AI companies and compensate for it.
CloudFlare is now making its market a reality.
It’s called Payment Per Crawl and CloudFlare will begin “experiment” in private beta on Tuesday. The owner of the experimental website can choose to scrape the site off the AI crawler individually, at a set rate. This is all “crawl” micropayments. Alternatively, website owners can choose to have AI crawlers scrape the site for free or block it entirely. CloudFlare claims that the tool allows website owners to check whether Crawler is scraping the site for AI training data, whether it appears in AI search responses, or for other purposes.
On a large scale, CloudFlare’s Marketplace is a big idea that can provide publishers with potential business models for the AI era. It also puts CloudFlare at the heart of everything. The launch of the market comes when news publishers face existential questions about how to reach their readers as Google’s search traffic disappears and AI chatbots become more popular.
There is no clear answer as to how news publishers will survive in the age of AI. Some, such as The New York Times, have filed lawsuits against high-tech companies to train AI models on news stories without permission. Meanwhile, other publishers have signed multi-year agreements to license AI model training content and bring content to the AI chatbot response.
Still, only large publishers have attacked AI licensing transactions, and it is still unclear whether they will provide a meaningful revenue stream. CloudFlare aims to create more durable systems that allow publishers to price their own terms.
The company announced Tuesday that a new website configured using CloudFlare will block all AI crawlers by default. Site owners must grant permission to access the site of a particular AI crawler. It says that with the CloudFlare change, all new domains give “default controls”.
Several large publishers, including Conde Nast, Time, The Associated Press, Atlantic, Adweek and Fortune, have signed on with CloudFlare to block AI crawling by default to support the company’s broader goal of “a permission-based approach to crawling.”
The business models that many of these publishers relied on for decades are gradually becoming less reliable. Historically, online publishers have allowed Google to scrape the site in exchange for referrals of Google searches translated into traffic to the site.
However, new data from CloudFlare suggests that Publishers may be worse in the AI era than in the Google search era. Some websites cite ChatGpt as the main traffic source, but that doesn’t seem to be the case in large part.
In June this year, CloudFlare says it found that every time Google’s Crawler introduced it, it had shattered its website 14 times. Meanwhile, Openai’s Crawler cut 17,000 websites for each referral, but humanity cut 73,000 times for each referral.
Meanwhile, Openai and Google are building AI agents designed to access websites on their behalf, collect information, and deliver them directly to users. The future where these tools are mainstream will have a major impact on publishers who rely on readers visiting the site.
CloudFlare points out that “true possibilities” of wages per crawl could emerge in the future of “agents.”
“What if the agent paywall can work perfectly programmatically at the network edge? Ask your favorite deep research program that will help you integrate the latest cancer research and legal briefs, or help you find the best restaurants in Soho and give that agent the budget to spend to get the best relevant content.”
To participate in the CloudFlare experimental market, both AI companies and publishers must set up with their CloudFlare accounts. With the account, both parties can set a price they wish to buy and sell “crawls” of publisher content. CloudFlare acts as an intermediary in these transactions, billing AI companies and distributing revenues to publishers.
CloudFlare spokesman Ripley Park told TechCrunch that despite many suggesting that digital currency is best for something like this, there are no stubcoins or cryptocurrencies associated with salary per crawl at this point.
CloudFlare’s marketplace feels like a bold vision for the future where many publishers and AI companies need to board. Still, there is no guarantee that the publisher will get a fair deal. Given that they are currently rubbing content for free, it can be difficult to convince AI companies to take part.
Nonetheless, CloudFlare appears to be one of the few companies in a position to make such a market a reality.
