mlsanityData to 7 Oct 2026

ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attribution

arXiv 2602.15769 ↗First posted 17 Feb 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal Large Language Models (mLLMs) are often used to answer questions in structured data such as tables in Markdown, JSON, and images. While these models can often give correct answers, users also need to know where those answers come from. In this work, we study structured data attribution/citation, which is the ability of the models to point to the specific rows and columns that support an answer. We evaluate several mLLMs across different table formats and prompting strategies. Our results show a clear gap between question answering and evidence attribution. Although question answering accuracy remains moderate, attribution accuracy is much lower, near random for JSON inputs, across all models. We also find that models are more reliable at citing rows than columns, and struggle more with textual formats than images. Finally, we observe notable differences across model families. Overall, our findings show that current mLLMs are unreliable at providing fine-grained, trustworthy attribution for structured data, which limits their usage in applications requiring transparency and traceability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 28 references that are in the map. First posted on arXiv 17 Feb 2026.

Builds on

22 of 28 references.

References · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  7. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  8. 2019Theoretical Limitations of Self-Attention in Neural Sequence Modelscited by 286
  9. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  10. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  11. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  12. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 117
10 more
  1. 2023DocLLM: A layout-aware generative language model for multimodal document understandingcited by 112
  2. 2022Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Modelscited by 108
  3. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 51
  4. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  5. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Modelscited by 34
  6. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 22
  7. 2024Can LLMs Evaluate Complex Attribution in QA? Automatic Benchmarking using Knowledge Graphscited by 16
  8. 2025HiBench: Benchmarking LLMs Capability on Hierarchical Structure Reasoningcited by 12
  9. 2025ConSens: Assessing context grounding in open-book question answeringcited by 4
  10. 2025On Synthesizing Data for Context Attribution in Question Answeringcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025MAVIS: A Benchmark for Multimodal Source Attribution in Long-form Visual Question Answering
  2. 2025Protecting multimodal large language models against misleading visualizations
  3. 2026VinQA: Visual Elements Interleaved Long-form Answer Generation for Real-World Multimodal Document QA
  4. 2026Efficient Table Retrieval and Understanding with Multimodal Large Language Models
  5. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectly
  6. 2026CiteVQA: Benchmarking Evidence Attribution for Trustworthy Document Intelligence
  7. 2024A Survey on Benchmarks of Multimodal Large Language Models
  8. 2025UPME: An Unsupervised Peer Review Framework for Multimodal Large Language Model Evaluation