Document QA
TAT-DQA
Visual document QA over financial-report pages that mix tables and text.
As of 13 August 2026. Held out. Zhu et al., arXiv 2207.11871. Official materials conflict between CC-BY-4.0 language and non-commercial-use language.
TAT-DQA is visual document QA on financial-report pages. Contexts are page images with tables and text, not the already-structured TAT-QA tables. Official exact match and official F1 remain separate. Human expert marks of 84.1 / 90.8 are the TAT-QA expert protocol reused as a ceiling, not a full TAT-DQA re-annotation.
Until the license conflict is cleared in writing, the pack cannot enter a commercial artifact. The figures below are the paper's 2022 systems, not a 2026 VLM ladder.
- Scale
- 16,558 questions over 3,067 pages from 2,758 documents
- Provenance
- Zhu et al., arXiv 2207.11871. Official materials conflict between CC-BY-4.0 language and non-commercial-use language.
- Contamination
- High. Public since 2022.
- Cannot establish
- Any commercial artifact until the license conflict is cleared in writing.
Metrics, kept separate
- Official exact match
- Official F1
- Numeric-tolerance accuracy
- Evidence recall
Paper table · cited paper · not an Antidote run · not a product score
Zhu et al., arXiv 2207.11871, Table 4, test split. Human expert is the TAT-QA expert protocol reported in the paper, not a full TAT-DQA re-annotation.
| System | Exact match | F1 |
|---|---|---|
| Human expert | 84.1% | 90.8% |
| MHST LayoutLMv2-L | 41.5% | 50.7% |
| MHST RoBERTa-L | 39.8% | 47.6% |
| TagOp | 33.7% | 42.5% |
| NumNet+ V2 | 30.6% | 40.1% |
Paper table · cited paper · not an Antidote run · not a product score
Zhu et al., arXiv 2207.11871, Table 5, LayoutLMv2-large. Arithmetic remains the weak F1 slice among typed items.
| Answer type | Exact match | F1 |
|---|---|---|
| Span | 41.1% | 58.3% |
| Counting | 43.2% | 43.2% |
| Arithmetic | 42.7% | 42.7% |
| Spans | 25.7% | 43.3% |
What the figures bound
MHST with LayoutLMv2-large reaches 41.5% exact match and 50.7% F1 on test. The text-only MHST variant is 39.8 / 47.6. TagOp is 33.7 / 42.5. NumNet+ V2 is 30.6 / 40.1. Human experts remain 84.1 / 90.8 on the reported cell.
By answer type, MHST test F1 is 58.3% on span items and 42.7% on arithmetic. The visual page is harder than TAT-QA's structured hybrid context. That is a 2022 bound, not a parse-and-answer product score.
What this page does not conclude
- A commercial run before written license clarification
- A 2026 frontier VLM ranking
- Hebbia or AlphaSense document-product performance
- An averaged EM-plus-F1 winner
Cited as calibration on these memos — not a score of those products.
Not attached to a software, advisory, or automation memo yet.