Legal retrieval
LegalBench-RAG
Exact-span passage retrieval over legal corpora sourced from CUAD, ContractNLI, MAUD, and PrivacyQA.
As of 13 August 2026. Conditional pack. Pipitone and Houir Alami, arXiv 2408.10343. Character precision and recall stay separate. The composite license chain is not approved for commercial use.
LegalBench-RAG asks a retriever for exact character spans, not document IDs and not generated answers. Precision at k and recall at k are different estimands. The paper's experiments run on LegalBench-RAG-mini, not the full 6,858-pair corpus.
Source datasets are CUAD, ContractNLI, MAUD, and PrivacyQA. Repository MIT does not license those sources. Until a source-by-source rights review passes, this pack cannot enter a commercial client or pitch artifact. The figures below are the paper's Recursive Character Text Splitter configuration without treating ALL as a winner.
- Scale
- 6,858 query/answer-span pairs over 714 documents. Paper experiments use LegalBench-RAG-mini.
- Provenance
- Pipitone and Houir Alami, arXiv 2408.10343. Repository MIT does not override CUAD, ContractNLI, MAUD, or PrivacyQA terms.
- Contamination
- High. Public since 2024. Queries and exact source spans are visible.
- Cannot establish
- Answer correctness, Harvey product performance, or a vendor moat. The pack stays blocked until the source license chain is approved.
Metrics, kept separate
- Character precision at k
- Character recall at k
- Character F1
- Recall@k
- nDCG@k
Paper table · cited paper · not an Antidote run · not a product score
Pipitone and Houir Alami, arXiv 2408.10343, Table 5. Recursive Character Text Splitter. Precision@1 and Recall@64 stay separate. Experiments are on the mini split.
| Source dataset | Precision@1 | Recall@1 | Recall@64 |
|---|---|---|---|
| PrivacyQA | 14.38% | 8.85% | 84.19% |
| ContractNLI | 6.63% | 7.63% | 61.72% |
| CUAD | 1.97% | 1.62% | 74.7% |
| MAUD | 2.65% | 1.65% | 28.28% |
What the figures bound
On Table 5, PrivacyQA Precision@1 is 14.38% and Recall@64 is 84.19%. MAUD is the hard slice: Precision@1 is 2.65% and Recall@64 is 28.28%. CUAD Recall@64 reaches 74.70% while Precision@1 is 1.97%. High recall at large k is not precise retrieval of the operative span.
Use the pack, after rights clearance, as a legal-retrieval diagnostic. Do not treat it as answer quality, as Harvey, or as a CLM moat.
What this page does not conclude
- Answer correctness or legal advice quality
- Harvey or any legal-product performance
- A commercial run before the license chain is approved
- A single LegalBench-RAG winner across k and datasets
Cited as calibration on these memos — not a score of those products.
Not attached to a software, advisory, or automation memo yet.