Reddit blocks Internet Archive to end sneaky AI scraping

Reddit is now blocking the Internet Archive (IA) from indexing popular Reddit threads after allegedly catching sneaky AI firms—restricted from scraping Reddit—instead simply scraping data from IA's archived content.

Where before IA's Wayback Machine dependably archived Reddit pages, profiles, and comments—as part of its mission to archive the Internet—moving forward, only screenshots of the Reddit homepage will be archived. As The Verge noted, this means the archive will only be useful as a snapshot of popular posts and news headlines each day, rather than providing a backup documenting deleted posts or a window into various Reddit subcultures or any given user's activity.

Reddit has not confirmed which AI firms were scraping its data from the Wayback Machine. The company's spokesperson, Tim Rathschmidt, would only confirm to Ars that Reddit has become "aware of instances where AI companies violate platform policies, including ours, and scrape data from the Wayback Machine."

Rathschmidt suggested there may be steps that IA could take to better defend against the AI scraping of archived Reddit content. That could perhaps lead Reddit to lift the restrictions on its scraping, which The Verge reported will be ramping up across Reddit starting today.

But Reddit also is taking this time to address other apparently longstanding privacy concerns, adding that restrictions are appropriate since the Wayback Machine problematically archives content that users have deleted.