General agents
GAIA
Tool-use, web, file, and reasoning questions that are easy for people and hard for assistants.
As of 13 August 2026. Held out. Mialon et al., arXiv 2311.12983. Gated terms. Never a public demo. Level scores stay separate.
GAIA is 466 assistant questions in three levels. Level 1 generally needs at most one tool. Level 3 needs longer plans. Accuracy by level is the load-bearing report. An overall rate that hides Level 3 at zero is refused here as a ranking device, even though the paper also states a 15% GPT-4-with-plugins figure against 92% for humans.
Gated Hugging Face terms block public redistribution of questions and attachments. Access-controlled internal use is possible only after terms approval. GPT-4 + plugins in Table 4 used manual plugin selection and is not a reproducible API cell.
- Scale
- 466 questions across three difficulty levels
- Provenance
- Mialon et al., arXiv 2311.12983. Gated Hugging Face terms; no clear SPDX data license.
- Contamination
- High. Public since 2023. Gated terms; do not reshare attachments.
- Cannot establish
- A public demo, a commercial client artifact, or a target-workflow score until gated terms are approved.
Metrics, kept separate
- Official accuracy by level
- Completion rate
- Tool calls
- Latency
Paper table · cited paper · not an Antidote run · not a product score
Mialon et al., arXiv 2311.12983, Table 4. Level scores stay separate. GPT-4 + plugins used manual plugin selection. Human is validation-annotator accuracy on valid questions.
| System | Level 1 | Level 2 | Level 3 |
|---|---|---|---|
| Human annotator | 93.9% | 91.8% | 87.3% |
| GPT-4 + plugins | 30.3% | 9.7% | 0% |
| AutoGPT (GPT-4) | 14.4% | 0.4% | 0% |
| GPT-4 Turbo | 13% | 5.5% | 0% |
| GPT-4 | 9.1% | 2.6% | 0% |
| Search engine | 7.4% | 0% | 0% |
What the figures bound
Human annotators are 93.9% on Level 1, 91.8% on Level 2, and 87.3% on Level 3. GPT-4 with plugins is 30.3%, 9.7%, and 0%. GPT-4 without plugins is 9.1%, 2.6%, and 0%. AutoGPT with a GPT-4 backend is 14.4% on Level 1 and 0.4% on Level 2.
The gap is the point of the pack: conceptually simple for people, still hard for 2023 assistants. It is not a diligence-workflow score and it is not a public exhibit of the questions themselves.
What this page does not conclude
- A public demo or redistributed question set
- A commercial client artifact before gated terms are approved
- Target-workflow or domain-product performance
- A single GAIA winner across levels
Cited as calibration on these memos — not a score of those products.
Not attached to a software, advisory, or automation memo yet.