Why AI researchers keep building something they think will kill humans
Why do AI researchers keep building technology they believe will kill? Inside the commercial, ethical, and psychological forces driving doomers.
TL;DR
- OpenAI's CFO revealed that their largest AI models can now train smaller ones, a form of recursive self-improvement (RSI).
- This capability is seen by some as a significant business opportunity and by others, particularly AI researchers, as a source of fear due to the rapid potential increase in AI power.
- Concerns about AI's potential to kill humans are being voiced by employees at companies like Anthropic, leading to significant debate in the AI community.
- Despite these fears, researchers continue development due to commercial incentives, the belief that they are in a race to develop AI safely, and recruitment strategies that frame AI safety as a world-saving mission.
- A perceived loss of agency among researchers, who feel they have little control over AI's direction despite awareness of its risks, also contributes to public warnings about existential threats.