Anthropic proposes three rules for measuring AI progress.
Anthropic's proposals, laid out in a big website, are:

TL;DR
- AI building the next version of itself is a key progress metric.
- Human oversight and intervention in AI actions are crucial.
- Resources powering AI development are vital for progress measurement.
- Anthropic has shared an internal snapshot of these metrics.