Amazon’s bet that AI benchmarks don’t matter

Posts from this topic will be added to your daily email digest and your homepage feed.

TL;DR

Amazon AI Chief Rohit Prasad believes AI benchmarks are flawed and not indicative of real-world utility.
He suggests that current benchmark evals are noisy and don't showcase the true capabilities of AI models.
Amazon has announced Nova Forge, a service designed to enable companies to train custom AI models.
Nova Forge provides access to Amazon's Nova model checkpoints at different training stages.
This service offers a solution to the challenges companies face when customizing AI models, such as limitations with closed models, risks with open-weight models, or the high cost of building from scratch.

Продолжить чтение оригинальной статьи

Yann LeCun reposted

Jean-Rémi King

💫Very happy to release NeuralBench, to benchmark Neuro AI models and datasets in the open!

🧵Thread, 💻Code, 📝White Paper below:

Sam Altman

i get some anxiety not using the smartest-available model/settings.

but sometimes i dont mind if it's really slow.

i wonder if we should focus more on a price/speed tradeoff relative to a price/intelligence tradeoff.

clem 🤗 reposted

Alok Bishoyi

one of the things i have been very curious about is how to measure and improve novelty for autoresearch run.

So introducing : @EVO__HQ 's autoresearch-novelty-bench : A benchmark for measuring whether AI agents during autoresearch runs propose genuinely novel ideas

built on top @PrimeIntellect's autonomous-speedrunning archive: claude Code & Codex racing on the modded-nanogpt speedrun. contains over 10k+ training runs, 600+ idea writeups, an intense 2-week burst of parallel autoresearch - we know what each agent tried, when, and whether it worked. thread

clem 🤗

Agree! was talking about this with @havoyan just a few days ago. That's also the reason why so much of the value has been accruing to the frontier models in my opinion (cc @GavinSBaker) because if you don't have the mental load to pick/change models, you'll just default to the one that has been marketed as "frontier" even if it's not the best trade-off for the majority of things you're doing with it (ex you can get cheaper more efficient models on telling you the weather, classifying, question answering,...)

clem 🤗

This graph captures what’s broken about AI evals: they structurally favor closed-source APIs that can route, fallback, ensemble, and optimize behind the scenes with no transparency.

No offense, @ArtificialAnlys, but how is comparing one model to two models fair? https://t.co/iuCu5GlWoQ

clem 🤗

To people in the answers saying "but opus 4.8 is weaker so without fallback, the score would even be higher": this is not necessarily true because of how any benchmark - which is an average of queries - work and what is called "the fallacy of division".

Even if Opus 4.8 has a lower average score on AA than Fable 5, it actually performs better than Fable 5 on some benchmark that compose the index of AA, especially where there's high refusal rate of Fable 5 (ex GPQA Diamond, AA-Omniscience). The same would go if you'd take a single benchmark btw as it's always an average of queries and the fact that a model has a higher score on average doesn't mean they answer better on 100% of queries.

So it's possible that Fable with Opus 4.8 fallbacks is getting a higher score than pure Fable, even if Opus 4.8 is weaker on average.

The challenge is no one knows, except the API provider, which is the challenge I'm pointing out.

More details below from Fable (or Opus?) themselves!

clem 🤗

“Cost per task varies by ~800x across models tested: Claude Fable 5 leads the benchmark but costs more than $31 per task on average, compared to ~$0.04 for DeepSeek V4 Flash (max). The strongest price/performance options are open weights models such as GLM-5.2 (max) and DeepSeek V4 Pro (max), with GLM-5.2 (max) scoring only ~90 Elo below Claude Opus 4.8 (max) for less than 25% of the cost”

Greg Brockman

benchmarks get saturated very quickly these days

Elon Musk

Interesting