Finance reasoning
FinanceBench
Open-book financial QA with human-adjudicated answers, evidence, and justifications.
As of 13 August 2026. Held out. Islam et al., arXiv 2311.11944. CC-BY-NC hosted metadata and no explicit upstream data license. Not a commercial published benchmark.
FinanceBench is open-book financial QA. The paper's public evaluation is 150 human-adjudicated cases, not the 10,231-question full set. Correct, incorrect, and refused are different outcomes. Incorrect is the dangerous class. They are not collapsed into accuracy.
Closed book, shared vector store, single vector store, long context, and oracle are different test conditions. Oracle is not a production retrieval setting. The pack stays out of client and pitch artifacts until written commercial and redistribution permission exists.
- Scale
- 10,231 questions; the paper evaluates a public sample of 150
- Provenance
- Islam et al., arXiv 2311.11944. Hosted metadata CC-BY-NC-4.0; upstream repository has no explicit data license.
- Contamination
- High. Public since 2023. The sample, answers, evidence, and model results are visible.
- Cannot establish
- Any commercial client or pitch claim until rights are cleared. A sample pass rate is not a workflow replacement.
Metrics, kept separate
- Human-adjudicated correctness
- Incorrect-answer rate
- Refusal rate
- Evidence recall
Paper table · cited paper · not an Antidote run · not a product score
Islam et al., arXiv 2311.11944, Table 2. Human-adjudicated labels on the public sample of 150. Incorrect is kept separate from refusal. Oracle is not a production retrieval setting.
| Configuration | Correct | Incorrect | Failed to answer |
|---|---|---|---|
| GPT-4-Turbo oracle | 85% | 15% | 0% |
| GPT-4-Turbo long context | 79% | 17% | 4% |
| Claude 2 long context | 76% | 21% | 3% |
| GPT-4-Turbo single store | 50% | 11% | 39% |
| Llama 2 single store | 41% | 54% | 5% |
| GPT-4-Turbo shared store | 19% | 13% | 68% |
| Llama 2 shared store | 19% | 70% | 11% |
| GPT-4-Turbo closed book | 9% | 3% | 88% |
What the figures bound
GPT-4-Turbo with a shared vector store is correct on 19% of the 150-case sample, incorrect on 13%, and refuses 68%. The paper's 81% figure is incorrect plus refused. A single-document vector store lifts correctness to 50%. Long context reaches 79% correct and still answers incorrectly 17% of the time. Oracle is 85% correct with a 15% incorrect remainder and no refusals.
Those cells are a 2023 diagnostic on a public sample. They are not a license to run the pack, not AlphaSense, and not a substitution case for an analyst process.
What this page does not conclude
- A commercial or pitch use of FinanceBench
- An AlphaSense or research-product score
- Replacement of a current financial-QA workflow
- A winner across retrieval configurations
Cited as calibration on these memos — not a score of those products.
Not attached to a software, advisory, or automation memo yet.