Contract analysis
ContractNLI
Hypothesis entailment, contradiction, or not-mentioned, with evidence spans.
As of 13 August 2026. Published benchmark. Koreeda and Manning, EMNLP 2021 findings. Contradiction is the dangerous class.
ContractNLI is a three-way label: entailment, contradiction, or not mentioned, plus evidence spans. Label accuracy can look strong while the class that creates legal risk stays weak. Span overlap is not advice quality.
Silence and contradiction are different misses. A system that treats them as the same error will look better than it is. Not-mentioned accuracy and span F1 belong in a diligence record as separate gates.
- Scale
- 607 contracts
- Provenance
- Stanford NLP ContractNLI, EMNLP 2021 findings.
- Contamination
- High. Public since 2021.
- Cannot establish
- Advice quality, negotiation outcome, or a CLM vendor moat.
Metrics, kept separate
- Label accuracy
- Evidence-span F1
- Not-mentioned accuracy
- Failure rate
Paper table · Koreeda and Manning, 2021 · not an Antidote run · not a product score
Koreeda and Manning, EMNLP 2021 findings, Table 3. Contradiction F1 stays much lower than entailment F1 and is not folded into accuracy.
| System | NLI accuracy | Entailment F1 | Contradiction F1 |
|---|---|---|---|
| DeBERTa v2 + CUAD | 89.2% | 85.9% | 40.5% |
| DeBERTa v2 xlarge | 88.5% | 85.5% | 36% |
| BERT-large | 87.5% | 83.4% | 35.7% |
| BERT-base | 83.8% | 76.5% | 28.7% |
What the figures bound
Span NLI BERT with DeBERTa v2 fine-tuned on CUAD reaches 89.2% NLI accuracy and 85.9% entailment F1. Contradiction F1 is 40.5%. The same pattern holds for DeBERTa v2 xlarge (88.5% accuracy, 36.0% contradiction F1) and BERT-large (87.5% / 35.7%). Accuracy is not a substitute for the weak class.
Use this pack to keep unsupported versus contradicted claims apart. Do not treat public NDAs as a target data room, and do not score a CLM or legal product from these cells.
What this page does not conclude
- Advice quality or negotiation outcome
- A CLM or legal-product score
- Span faithfulness on client matter documents
- A vendor moat