# Where is this coming from? Making groundedness count in the evaluation of Document VQA models

arXiv: https://arxiv.org/abs/2503.19120 · HTML: https://mlsanity.com/paper/2503.19120/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 24 Mar 2025.

## Builds on (most cited first, 15 of 15)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suites](https://mlsanity.com/paper/2404.16821/) (2024)
- [When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards](https://mlsanity.com/paper/2402.01781/) (2024)
- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation](https://mlsanity.com/paper/2205.09921/) (2022)
- [ICDAR 2019 Competition on Scene Text Visual Question Answering](https://mlsanity.com/paper/1907.00490/) (2019)
- [Document Collection Visual Question Answering](https://mlsanity.com/paper/2104.14336/) (2021)
- [Grounding Language Model with Chunking-Free In-Context Retrieval](https://mlsanity.com/paper/2402.09760/) (2024)
- [Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts](https://mlsanity.com/paper/2312.00968/) (2023)
- [Reading Order Matters: Information Extraction from Visually-rich Documents by Token Path Prediction](https://mlsanity.com/paper/2310.11016/) (2023)
- [ROPE: Reading Order Equivariant Positional Encoding for Graph-based Document Information Extraction](https://mlsanity.com/paper/2106.10786/) (2021)
- [DocGraphLM: Documental Graph Language Model for Information Extraction](https://mlsanity.com/paper/2401.02823/) (2024)
- [TreeForm: End-to-end Annotation and Evaluation for Form Document Parsing](https://mlsanity.com/paper/2402.05282/) (2024)
- [Arctic-TILT. Business Document Understanding at Sub-Billion Scale](https://mlsanity.com/paper/2408.04632/) (2024)

## Built on by (most cited first, 4 of 4)

- [CoCoLex: Confidence-guided Copy-based Decoding for Grounded Legal Text Generation](https://mlsanity.com/paper/2508.05534/) (2025)
- [Towards Reliable and Interpretable Document Question Answering via VLMs](https://mlsanity.com/paper/2509.10129/) (2025)
- [ExStrucTiny: A Benchmark for Schema-Variable Structured Information Extraction from Document Images](https://mlsanity.com/paper/2602.12203/) (2026)
- [Amnesia by Design, Memory By Necessity: Persistent State for Document Intelligence](https://mlsanity.com/paper/2609.32041/) (2026)

## Abstract (arXiv)

Document Visual Question Answering (VQA) models have evolved at an impressive rate over the past few years, coming close to or matching human performance on some benchmarks. We argue that common evaluation metrics used by popular benchmarks do not account for the semantic and multimodal groundedness of a model's outputs. As a result, hallucinations and major semantic errors are treated the same way as well-grounded outputs, and the evaluation scores do not reflect the reasoning capabilities of the model. In response, we propose a new evaluation methodology that accounts for the groundedness of predictions with regard to the semantic characteristics of the output as well as the multimodal placement of the output within the input document. Our proposed methodology is parameterized in such a way that users can configure the score according to their preferences. We validate our scoring methodology using human judgment and show its potential impact on existing popular leaderboards. Through extensive analyses, we demonstrate that our proposed method produces scores that are a better indicator of a model's robustness and tends to give higher rewards to better-calibrated answers.