tech
Cloudflare is cracking down on “multi-purpose” crawlers.
Starting September 15th, Cloudflare will block bots that scrape ad-supported websites for search indexing and AI training at the same time. The goal is to encourage AI companies to separate their crawlers for different purposes, giving publishers more control over which ones to allow into their websites.
TL;DR
- Cloudflare will block multi-purpose crawlers starting September 15th.
- These crawlers scrape ad-supported websites for both search indexing and AI training.
- The goal is to encourage AI companies to separate their crawling functions.
- This change aims to give website publishers more control over data access.