tech
An Anthropic researcher just gave us a peek at self-improving AI
Given 10 benchmarks for specific misaligned behaviors, the automated systems were able to improve performance on every single one without degrading overall performance.
TL;DR
- Anthropic researchers have developed an Automated Alignment Researcher (AAR) system.
- The AAR system successfully improved AI model performance on 10 alignment benchmarks.
- Automated researchers replicated traditional research steps: literature search, method proposal, and model training.
- The AAR system outperformed experienced humans in alignment research, both in effectiveness and cost ($4/hour vs. $150/hour).
- Limitations include the dependency on accurate benchmarks and the need for ongoing maintenance of literature and benchmarks.