Legal reasoning
LegalBench
162 legal reasoning tasks spanning issue-spotting, rule recall, application, conclusion, interpretation, and rhetorical understanding.
As of 13 August 2026. Conditional pack. Guha et al. 2023. Per-task licenses are not inherited from the repository. A single LegalBench macro score is refused.
LegalBench is 162 tasks from 36 sources spanning issue-spotting, rule recall, application, conclusion, interpretation, and rhetorical understanding. Category scores stay separate because they are different skills. Interpretation includes CUAD-style clause classification, which is easier than span extraction.
Rule-application is graded twice: correctness and analysis. A right outcome without the inference is a different product failure than a wrong outcome. Slice tables with n of 12 to 29 are maps of where models diverge, not population estimates. GPT-4, GPT-3.5, and Claude-1 here are 2023 retrievals, not a 2026 frontier ladder.
- Scale
- 162 tasks from 36 sources, including CUAD-derived clause classification
- Provenance
- Guha et al. 2023, Stanford LegalBench site, arXiv 2308.11462. Task licenses are not inherited from the repository.
- Contamination
- High. Public since 2023. Per-task licenses are composite and must be approved before a run.
- Cannot establish
- Legal correctness in a client matter, Harvey product performance, or a vendor moat. A single LegalBench macro score is refused.
Metrics, kept separate
- Balanced accuracy by reasoning category
- Rule-application correctness
- Rule-application analysis
- Task-specific official metrics
Paper table · Guha et al., 2023 · not an Antidote run · not a product score
Guha et al., 2023, Table 2. Balanced accuracy within each reasoning category. GPT-4, GPT-3.5 (text-davinci-003), and Claude-1, retrieved May to August 2023. Not a single LegalBench winner score.
| Reasoning category | GPT-4 | GPT-3.5 | Claude-1 |
|---|---|---|---|
| Issue-spotting | 82.9% | 60.9% | 58.1% |
| Rule recall | 59.2% | 46.3% | 57.7% |
| Rule conclusion | 89.9% | 78% | 79.5% |
| Interpretation | 75.2% | 72.6% | 67.4% |
| Rhetorical | 79.4% | 66.7% | 68.9% |
Paper table · Guha et al., 2023 · not an Antidote run · not a product score
Guha et al., 2023, Table 3. Correctness and analysis are graded separately. A correct outcome without analysis is a different failure than an incorrect outcome.
| System | Correctness | Analysis |
|---|---|---|
| GPT-4 | 82.2% | 79.7% |
| Claude-1 | 61.4% | 59% |
| GPT-3.5 | 58.5% | 44.2% |
Paper table · Guha et al., 2023 · not an Antidote run · not a product score
Guha et al., 2023, Table 6. Slice n is 12 to 29. These percentages show where the models diverge, not a population estimate.
| Slice | n | GPT-3.5 | GPT-4 |
|---|---|---|---|
| Non-assertive conduct | 19 | 100% | 100% |
| In-court statement | 14 | 57% | 93% |
| Standard hearsay | 29 | 97% | 97% |
| Non-verbal hearsay | 12 | 33% | 75% |
| Not for truth | 20 | 25% | 45% |
What the figures bound
GPT-4 is 82.9% on issue-spotting and 59.2% on rule recall. Rule conclusion is 89.9%; interpretation 75.2%; rhetorical understanding 79.4%. GPT-3.5 and Claude-1 are close to each other on several categories and far from GPT-4 on issue-spotting. Rule-application correctness is 82.2% for GPT-4 and 58.5% for GPT-3.5; analysis is 79.7% and 44.2%.
Hearsay slices show the same refusal to average: GPT-4 lifts non-verbal hearsay from 33% to 75% and in-court statements from 57% to 93%, while both models remain weak when the statement is not offered for its truth (25% vs 45%). These are 2023 model diagnostics. They do not score Harvey, and they do not decide legal correctness in a client matter.
What this page does not conclude
- Legal correctness on a client matter
- Harvey or any legal-product performance
- A 2026 frontier ranking
- A LegalBench macro winner