tech
Cloudflare's latest AI rankings expose the web's biggest free rider
Anthropic's web scraping practices spotlight ethical AI dilemmas, with a high crawl-to-refer ratio undermining the digital content economy.
TL;DR
- Anthropic exhibits the highest crawl-to-refer ratio among major AI companies, crawling significantly more than it refers traffic.
- This practice by AI companies is seen as 'strip-mining' the web and breaking the historical economic bargain that powered the internet.
- AI answer engines and chatbots providing direct answers reduce user visits to original content websites, diminishing financial incentives for content creation.
- Anthropic's ethical claims are contrasted with its data-gathering practices, which some compare to 'distillation attacks' it criticizes in rivals.
- DuckDuckGo is noted as one of the few players maintaining a more balanced crawl-to-refer ratio.