Finance retrieval
FinAgentBench
Agentic retrieval over SEC filings: rank document type, then rank passages inside the selected filing.
As of 13 August 2026. Conditional pack. Choi et al., arXiv 2508.14052. Document-type ranking and chunk ranking stay separate. Pin the hosted revision before a run.
FinAgentBench scores agentic retrieval in two stages. The first ranks five SEC document types for a query. The second ranks paragraph-level chunks inside the selected filing. nDCG, MAP, and MRR at 5 are different ranking estimands. They are not averaged into one retrieval score.
The paper evaluates GPT-o3, Claude Opus 4, and Claude Sonnet 4 in a retrieval-only setting, with no external tools. A later fine-tune of GPT-o4-mini is a separate cell. Paper counts and the hosted Hugging Face release can differ; the acquired revision is the run authority.
- Scale
- About 26,000 expert-annotated examples on S&P 500 firms. Paper and hosted counts differ; pin the revision.
- Provenance
- Choi et al., arXiv 2508.14052. Hosted dataset MIT at the observed revision; recheck before acquisition.
- Contamination
- Moderate. Public since 2025. Recheck the hosted revision, fingerprint, and SEC-filing provenance before a run.
- Cannot establish
- Answer correctness on a live research workflow, AlphaSense product performance, or economic value.
Metrics, kept separate
- Document-type nDCG@5
- Document-type MAP@5
- Document-type MRR@5
- Chunk nDCG@5
- Chunk MAP@5
- Chunk MRR@5
Paper table · cited paper · not an Antidote run · not a product score
Choi et al., arXiv 2508.14052, Table 1. nDCG@5, MAP@5, and MRR@5 on five SEC document types. Not averaged.
| System | nDCG@5 | MAP@5 | MRR@5 |
|---|---|---|---|
| Claude Sonnet 4 | 0.783 | 0.849 | 0.892 |
| Claude Opus 4 | 0.773 | 0.84 | 0.875 |
| GPT-o3 | 0.77 | 0.829 | 0.875 |
Paper table · cited paper · not an Antidote run · not a product score
Choi et al., arXiv 2508.14052, Table 2. Passage ranking inside the selected filing. Weaker than document-type ranking on every system.
| System | nDCG@5 | MAP@5 | MRR@5 |
|---|---|---|---|
| Claude Sonnet 4 | 0.419 | 0.296 | 0.567 |
| Claude Opus 4 | 0.418 | 0.307 | 0.568 |
| GPT-o3 | 0.351 | 0.257 | 0.538 |
What the figures bound
Document-type ranking is relatively strong. Claude Sonnet 4 reaches 0.783 nDCG@5, 0.849 MAP@5, and 0.892 MRR@5. Chunk ranking is much weaker: the same model is 0.419 nDCG@5. Claude Opus 4 leads chunk MAP@5 at 0.307. Knowing which filing type to open is not the same as finding the passage.
These cells bound 2025 commercial models on public S&P 500 filings. They do not score AlphaSense, they do not measure answer correctness, and they do not establish a research-product moat.
What this page does not conclude
- Answer correctness on a live research question
- An AlphaSense or Hebbia product score
- A single agentic-retrieval winner
- Operating economics of a filing-research workflow