Pay up or stop scraping: Cloudflare program charges bots for each crawl

Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites.

In a blog Tuesday, Cloudflare explained that its "pay-per-crawl" feature is currently in a private beta. A small number of publishers and content creators will participate in the experiment. Each publisher will be able to set their own prices that bots must pay before scraping content, Cloudflare said.

Matthew Prince, CEO of Cloudflare, said the feature would ensure that the Internet as we know it will survive "the age of AI."

"Original content is what makes the Internet one of the greatest inventions in the last century, and it's essential that creators continue making it," Prince said. "AI crawlers have been scraping content without limits. Our goal is to put the power back in the hands of creators, while still helping AI companies innovate. This is about safeguarding the future of a free and vibrant Internet with a new model that works for everyone."

Some participating publishers expressed optimism in the press release that Cloudflare's pay-per-crawl feature could potentially stop the endless scraping that publishers defending copyrights have alleged represents wide-scale theft.

Any content creators interested in joining the beta can sign up, Cloudflare noted, and perhaps eventually, they too can "be compensated for their contributions to the AI economy."

In the meantime, only the publishers involved in the beta will be able to choose which bots can access which parts of their sites, experimenting with blocking all bots or allowing certain bots to access certain content.

Cloudflare's program also gives them the flexibility to charge some bots while letting other bots scrape for free. This lets publishers that have negotiated deals with AI companies to allow approved scraping while still protecting their content from companies that have not yet struck licensing deals.