Evals & benchmarks

Public tests that bound generic capability.

These pages collect published tests that bound what a generic system already does on a task type. They are not Antidote-measured scores of current models, and they are not scores of the products named on the memos. Paper tables stay labeled as paper tables. Third-party leaderboards stay labeled as third-party leaderboards. Some datasets need a license before use. Others stay out. There is no composite winner ranking.

Default workflow diagnostics

Public tests that bound a task type used on the memos. They are calibration, not a product score.

Contract analysis

ContractEval

Clause-level legal-risk identification over 41 risk categories, including a laziness metric for answering that no related clause exists when one does.

License-gated. Methodology and results paper over CUAD.

Contract analysis

ContractNLI

Hypothesis entailment, contradiction, or not-mentioned, with evidence spans.

Published benchmark. 607 contracts

Finance reasoning

ConvFinQA

Conversational numerical reasoning over financial reports, with history across turns.

Published benchmark. 3,892 conversations, 14,115 questions

Contract analysis

CUAD

Extractive clause location across 41 commercial-contract categories.

Published benchmark. 510 contracts, more than 13,000 expert annotations

Finance retrieval

FinAgentBench

Agentic retrieval over SEC filings: rank document type, then rank passages inside the selected filing.

License-gated. About 26,000 expert-annotated examples on S&P 500 firms. Paper and hosted counts differ; pin the revision.

Finance reasoning

FinQA

Numerical reasoning as executable programs over financial reports.

Published benchmark. About 8,000 questions over about 2,800 reports

Legal reasoning

LegalBench

162 legal reasoning tasks spanning issue-spotting, rule recall, application, conclusion, interpretation, and rhetorical understanding.

License-gated. 162 tasks from 36 sources, including CUAD-derived clause classification

Document parsing

ParseBench

Deterministic PDF parsing across tables, charts, content faithfulness, semantic formatting, and visual grounding.

Published benchmark. 2,078 unique pages, 1,211 documents, 169,011 deterministic rules

Finance reasoning

TAT-QA

Hybrid table and text questions on financial reports.

Published benchmark. 16,552 questions over 2,757 hybrid contexts

Frontier baselines and composites (not product scores)

General, occupation, and composite ladders. Useful as context. Not a diligence grade and not a vendor ranking.

Knowledge work

AA Intelligence Index

Artificial Analysis nine-eval composite: GDPval-AA v2, Tau3-Banking, Terminal-Bench v2.1, SciCode, HLE, GPQA Diamond, CritPt, AA-Omniscience, AA-LCR.

License-gated. AA Intelligence Index v4.1.1, observed 13 August 2026

Knowledge work

AA-Briefcase

Realistic multi-file knowledge-work projects (slides, spreadsheets, memos) scored on rubric, analytical quality, and presentation.

License-gated. Artificial Analysis Briefcase harness, observed Jul 2026

General baseline

BIG-bench

200-plus tasks designed to exceed model capability at release. Useful for finding remaining failure types, not average workflow performance.

License-gated. 200-plus tasks. Data is not a single parquet download.

Search agents

BrowseComp

Hard-to-find, short-answer facts that require persistent web browsing.

License-gated. 1,266 questions with encrypted short answers

Knowledge work

Coding agents

Agentic coding with a named harness: Terminal-Bench, DeepSWE, and SWE-Atlas-QnA as reported by Artificial Analysis.

License-gated. Artificial Analysis Coding Agent Index, observed Jul 2026

Finance reasoning

FinanceQA

Derive ratios and performance metrics from annual reports and statements rather than quote a stated figure.

License-gated. About 4,000 entries. Gated on Hugging Face.

Financial NLP

Financial News Multisource

News at volume for monitoring, language modelling, and signal extraction rather than document QA.

License-gated. Multi-source financial news spanning 1990 to 2025. Pin a sample subset at acquisition.

Financial NLP

Financial PhraseBank

Three-class sentiment on financial news sentences annotated by finance readers.

License-gated. 4,845 sentences

Finance reasoning

Financial QA 10K

Locate and interpret disclosures in 10-K narrative sections (MD&A, risk factors, business).

License-gated. About 7,000 QA pairs. Pin the hosted revision at acquisition.

Finance agents

FinGAIA

Chinese-language multi-file, multi-tool, multi-step financial agent workflows.

License-gated. Public questions as a single workbook plus attachments. Pin the hosted revision at acquisition.

Financial NLP

FinGPT Sentiment

Instruction-tuning financial news and social-media sentiment. Used to ask whether a small fine-tune matches a frontier model on tone.

License-gated. FinGPT sentiment instruction set. Pin the hosted revision at acquisition.

Document QA

FinMMDocR

Multi-step numerical reasoning over financial page images where the evidence sits in the image, not a handed text layer.

License-gated. AAAI 2026 multimodal financial-document benchmark. Images and text ship in separate folders.

Finance retrieval

FinRetrieval

Retrieve specific numeric values from a structured financial database versus web search.

License-gated. 500 questions in the public Daloopa FinRetrieval release

Knowledge work

GDPval

Economically valuable knowledge-work tasks across occupations, scored as win rate or Elo against a human or model reference.

License-gated. OpenAI GDPval occupations; Artificial Analysis reports a GDPval-AA v2 harness

Frontier baseline

GPQA Diamond

Graduate-level science questions designed to be hard for non-experts and still hard for models.

License-gated. GPQA Diamond split. Fable 5 can fall back to Opus 4.8 on scientific items in the AA harness.

General baseline

HELM

Holistic evaluation design: accuracy, calibration, robustness, fairness, and efficiency measured together rather than one headline score.

License-gated. Stanford CRFM HELM scenarios. A runnable framework, not a single download.

Frontier baseline

Humanity's Last Exam

Closed-ended expert questions designed to be hard for current models.

License-gated. Public HLE set; Artificial Analysis reports Fable 5 with Opus 4.8 fallback on a share of tasks

Legal retrieval

LegalBench-RAG

Exact-span passage retrieval over legal corpora sourced from CUAD, ContractNLI, MAUD, and PrivacyQA.

License-gated. 6,858 query/answer-span pairs over 714 documents. Paper experiments use LegalBench-RAG-mini.

Deep research

LiveDRBench

Open-web research scored as claim precision, recall, and F1, not as a long-form report grade.

License-gated. 100 tasks across eight categories, originally grounded in May to June 2025 web state

General baseline

MMLU

Broad academic multiple-choice across 57 subjects. A general baseline, not a professional-document workflow.

License-gated. About 14,000 questions

Held out — no commercial use

License, permission, or map packs that stay out of commercial artifacts. No score chart is shown as if it were a product result.

Analytics agents

AIDABench

Spreadsheet, report, chart, and file-generation analytics tasks with separate evaluators.

Not included. The paper reports a best pass-at-1 of 59.4%

Finance reasoning

FinanceBench

Open-book financial QA with human-adjudicated answers, evidence, and justifications.

Not included. 10,231 questions; the paper evaluates a public sample of 150

General agents

GAIA

Tool-use, web, file, and reasoning questions that are easy for people and hard for assistants.

Not included. 466 questions across three difficulty levels

Finance landscape

Open FinLLM Leaderboard

A map of 42 financial datasets across seven categories. Use it to find underlying packs, not as a single score.

License-gated. FINOS Foundation leaderboard. Live index, not a downloadable eval set.

Analytics agents

SpreadsheetBench

Real spreadsheet QA plus multi-sheet editing, debugging, and visualization.

Not included. Forum-derived workbooks; pin the verified subset at acquisition

Document QA

TAT-DQA

Visual document QA over financial-report pages that mix tables and text.

Not included. 16,558 questions over 3,067 pages from 2,758 documents