As AI data scrapers steal revenue from websites, some are fighting back

Swarms of AI “crawlers” are rampaging the internet, scouring billions of websites for data to feed big tech companies’ algorithms, all without permission or payment, upending the online economy.

Before the rise of AI chatbots, websites allowed search engines to access their content in exchange for increased visibility, a system that generated traffic and advertising revenue.

However, rapid developments in generative AI have enabled tech giants like Google and OpenAI to use web crawlers to gather information for chatbots without humans ever having to visit the original site.

Traditional content production companies such as media outlets are being overtaken by AI crawlers, putting pressure on online operations and advertising revenue.

“Sites that allowed bots to access their content were getting readers in exchange for readers,” said Kurt Muehmel, head of AI strategy at data management company Dataiku.

But with the advent of generative AI, that model “completely broke down,” he told AFP.

Human internet traffic to Wikipedia will drop by 8 percent between 2024 and 2025 due to the rise in AI search engine abstracts, the online encyclopedia reported last month.

“The fundamental tension is that the new business of the AI-driven internet is not going to generate traffic,” said Matthew Prince, CEO of US internet service provider Cloudflare.

“No trespassing”

Cloudflare, which processes more than 20% of all internet traffic, announced new measures this summer aimed at stopping AI crawlers from accessing content without payment or permission from website owners.

“It’s basically like putting up a speed limit sign or a no trespassing sign,” Prince told AFP on the sidelines of the Web Summit in Lisbon.

“A badly behaved bot can get through it, but we can track it…Over time, we can tighten those controls in a way that we’re confident AI companies can’t break through.”

The measure applies to more than 10 million websites and has already “attracted the attention of artificial intelligence giants,” he added.

TollBit, a small American startup, provides online news publishers with tools to block, monitor and monetize AI crawler traffic.

CEO and co-founder Toshit Panigrahi said, “The Internet is a highway,” and described the company as a “tollbooth on the Internet.”

TollBit works with more than 5,600 sites, including US Today, Time Magazine, and the Associated Press, allowing news organizations to set their own access fees for their content.

While the analysis is free for publishers, AI companies are charged a “transaction fee for each piece of content they access.”

But for Meumel, online takeover by AI crawlers cannot be resolved with “partial measures or individual companies.”

“This is an evolution of the entire internet economy, and it’s going to take years,” he said.

If swarms of bots continue to roam freely online, “all the incentive to create content will disappear,” Prince said.

“That’s a loss not only for us humans who want to consume content, but also for AI companies who need original content to train their systems.”

© 2025 AFP