Cloudflare is now experimenting with tools that will allow content creators to charge a fee to AI crawlers to scrape their websites.
In a blog Tuesday, Cloudflare explained that its „pay-per-crawl“ feature is currently in a private beta. A small number of publishers—including AdWeek, The Associated Press, The Atlantic, BuzzFeed, Fortune, Gannett, and Ars Technica owner Condé Nast—will participate in the experiment. Each publisher will be able to set their own prices that bots must pay before scraping content, Cloudflare said.
Matthew Prince, CEO of Cloudflare, said the feature would ensure that the Internet as we know it will survive „the age of AI.“