Contract analysis
CUAD
Extractive clause location across 41 commercial-contract categories.
As of 13 August 2026. Published benchmark. Hendrycks et al., 2021, The Atticus Project. Clause location is not legal operability.
CUAD is extractive clause location across 41 commercial-contract categories. Token overlap and exact span match answer where a clause lives. They do not decide whether the clause is legally operative in a client fact pattern.
AUPR and precision at 80% recall are different estimands. A model can rank higher on one and lower on the other. Averaging them manufactures a ranking that neither metric supports. Critical misses and no-answer accuracy stay separate in a diligence run.
- Scale
- 510 contracts, more than 13,000 expert annotations
- Provenance
- The Atticus Project CUAD, dataset card CC-BY-4.0.
- Contamination
- High. Public since 2021 and widely mirrored.
- Cannot establish
- Legal correctness in a client fact pattern, or product defensibility.
Metrics, kept separate
- Token F1
- Exact span match
- Macro by clause type
- No-answer accuracy
- Critical misses
Paper table · Hendrycks et al., 2021 · not an Antidote run · not a product score
Hendrycks et al., 2021, Table 2. AUPR and precision at 80% recall are different estimands and are not averaged.
| System | AUPR | Precision @ 80% recall | Precision @ 90% recall |
|---|---|---|---|
| DeBERTa-xlarge | 47.8% | 44% | 17.8% |
| RoBERTa-large | 48.2% | 38.1% | 0% |
| RoBERTa-base | 42.6% | 31.1% | 0% |
| ALBERT-xxlarge | 38.4% | 31% | 0% |
| BERT-base | 32.4% | 8.2% | 0% |
What the figures bound
DeBERTa-xlarge has 47.8% AUPR and 44.0% precision at 80% recall. RoBERTa-large has a higher AUPR (48.2%) and worse precision at 80% recall (38.1%). At 80% recall, even the stronger precision figure still means roughly two irrelevant hits per relevant clause. Precision at 90% recall collapses further (17.8% for DeBERTa-xlarge; 0% in the reported RoBERTa-large cell).
That is a bound on 2021 extractive models on public CUAD contracts. It is not legal advice quality, not Harvey or Hebbia, and not a CLM moat.
What this page does not conclude
- Legal correctness in a client fact pattern
- A product or vendor score
- Negotiation outcome or reviewer minutes
- A single CUAD winner from mixed estimands