tech

April 7, 2026

Testing suggests Google's AI Overviews tells millions of lies per hour

Is 90 percent accuracy good enough for a search robot?

Testing suggests Google's AI Overviews tells millions of lies per hour

TL;DR

  • Google's AI Overviews, powered by Gemini, has faced criticism for accuracy issues since its launch.
  • A New York Times analysis, with help from Oumi, found AI Overviews to be accurate 90% of the time.
  • This means approximately 1 in 10 AI Overviews provide incorrect information.
  • The testing utilized the SimpleQA evaluation, a common benchmark for generative AI factuality.
  • Google disputes the study's findings, claiming the SimpleQA test contains incorrect information and doesn't reflect real user searches.
  • The accuracy rate improved from 85% to 91% after Gemini 3 updates, but the error rate remains a concern.
  • Examples of inaccuracies include providing wrong dates for historical events and wrongly stating the existence of the Classical Music Hall of Fame.
  • Google uses different AI models for queries, including faster, less expensive 'Flash' models, which may impact accuracy.
  • Despite grounding AI with web data improving accuracy, users are encouraged to verify AI-generated summaries.
  • The inherent non-deterministic nature of generative AI makes consistent verification challenging.

Continue reading the original article

Made withNostr