WANDR Benchmark: Evaluating Research Agents That Must Search Wide and Deep
A benchmark for high-volume, evidence-heavy knowledge work.
TL;DR
- WANDR is a new open benchmark and evaluation harness for knowledge work, designed to test AI agents on tasks requiring both wide discovery (many entities) and deep investigation (thorough evidence).
- The benchmark comprises 500 realistic tasks, ranging from dozens to thousands of verifiable records, simulating jobs like competitive mapping and market analysis.
- Current AI systems struggle significantly with wide-and-deep research, with top performers achieving only 0.363 soft F1 and 0.133 hard F1 scores, indicating a long way to go for complete coverage and evidence verification.
- WANDR uses a flexible 'qualification key hierarchy' to structure tasks, allowing for validation of independent paths through nested entities and evidence.
- Evaluation is reference-free, focusing on verifying cited evidence for each claim, with scores differentiating between partial progress (soft F1) and complete coverage (hard F1).
- Key findings show that partial progress is common but complete coverage is rare, scale exacerbates problems, and deeper hierarchies significantly increase failure rates.
- The dominant challenges are discovery and evidence construction, with systems failing to meet substantive task requirements or fully support claims with excerpts.
- Search as Code (SaC) systems show promise due to their ability to express complex logic programmatically, aligning well with the benchmark's demands.