16 Million Stolen Conversations, 24,000 Fake Accounts, and What It Means for the AI Tools You're Choosing This Week + The Manifold Probe
Three Chinese AI labs just ran 16 million conversations with Claude through 24,000 fake accounts to steal its capabilities. The headlines are calling it a cold war. That framing is wrong — and if you stop there, you’ll miss what this actually means for every AI tool your team is using right now.
TL;DR
- The framing of Chinese AI labs stealing capabilities as a 'cold war' is inaccurate and obscures the economic forces driving model distillation.
- Distillation is characterized as piracy, similar to historical piracy waves, which tend to slow down but not stop.
- Distilled models exhibit failures in sustained, autonomous, agentic work that are not measured by current benchmarks.
- The incentive to extract frontier AI capabilities is universal, not limited to state-backed entities.
- The Manifold Probe is presented as a structured instrument to test and compare AI models based on custom stress-tests derived from actual work, rather than synthetic benchmarks.