mlsanityData to 14 Oct 2026

Multimodal Table Understanding

arXiv 2406.08100 ↗First posted 12 Jun 2024cs.CL cs.AIcited by 44

Not on the map. Why?

Abstract (arXiv)

Although great progress has been made by previous table understanding methods including recent approaches based on large language models (LLMs), they rely heavily on the premise that given tables must be converted into a certain text sequence (such as Markdown or HTML) to serve as model input. However, it is difficult to access such high-quality textual table representations in some real-world scenarios, and table images are much more accessible. Therefore, how to directly understand tables using intuitive visual information is a crucial and urgent challenge for developing more practical applications. In this paper, we propose a new problem, multimodal table understanding, where the model needs to generate correct responses to various table-related requests based on the given table image. To facilitate both the model training and evaluation, we construct a large-scale dataset named MMTab, which covers a wide spectrum of table images, instructions and tasks. On this basis, we develop Table-LLaVA, a generalist tabular multimodal large language model (MLLM), which significantly outperforms recent open-source MLLM baselines on 23 benchmarks under held-in and held-out settings. The code and data is available at this https://github.com/SpursGoZmy/Table-LLaVA

Key facts. Cited by 44 papers in the map (data through 14 Oct 2026). Builds on 41 of its 60 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

41 of 60 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  9. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  10. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
29 more
  1. 2023LIMA: Less Is More for Alignmentcited by 1,100
  2. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,076
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  5. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  6. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 434
  7. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  8. 2021TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Financecited by 408
  9. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 357
  10. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  11. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  12. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  13. 2019Image-based table recognition: data, model, and evaluationcited by 237
  14. 2021TAPEX: Table Pre-training via Learning a Neural SQL Executorcited by 235
  15. 2017Challenges in Data-to-Document Generationcited by 226
  16. 2021FeTaQA: Free-form Table Question Answeringcited by 175
  17. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  18. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 137
  19. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  20. 2022MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuningcited by 109
  21. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 86
  22. 2023Vary: Scaling up the Vision Vocabulary for Large Vision-Language Modelscited by 84
  23. 2021AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industrycited by 44
  24. 2024Small Language Model Meets with Reinforced Vision Vocabularycited by 43
  25. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  26. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 36
  27. 2016TabMCQ: A Dataset of General Knowledge Tables and Multiple-choice Questionscited by 18
  28. 2023An Empirical Study of Instruction-tuning Large Language Models in Chinesecited by 17
  29. 2023From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuningcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 24; 2026: 13.

44 papers.

Sort:

By area:

  1. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  2. 2024OCRBench v2: An Improved Benchmark for Evaluating Large Multimodal Models on Visual Text Localization and Reasoningcited by 173
  3. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  4. 2025Lumina-DiMOO: An Omni Diffusion Large Language Model for Multi-Modal Generation and Understandingcited by 88
  5. 2024Baichuan-Omni Technical Reportcited by 42
  6. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 28
  7. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  8. 2025OmniCaptioner: One Captioner to Rule Them Allcited by 20
  9. 2024Tabular Data Augmentation for Machine Learning: Progress and Prospects of Embracing Generative AIcited by 15
  10. 2025Multimodal Tabular Reasoning with Privileged Structured Informationcited by 15
  11. 2025Benchmarking Multimodal LLMs on Recognition and Understanding over Chemical Tablescited by 13
  12. 2024Chimera: Improving Generalist Model with Domain-Specific Expertscited by 11
32 more
  1. 2025Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflowscited by 10
  2. 2024HumanVLM: Foundation for Human-Scene Vision-Language Modelcited by 8
  3. 2025NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tablescited by 8
  4. 2025Megrez-Omni Technical Reportcited by 7
  5. 2025FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agentscited by 7
  6. 2025TableMoE: Neuro-Symbolic Routing for Structured Expert Reasoning in Multimodal Table Understandingcited by 7
  7. 2025HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimizationcited by 6
  8. 2025TABLET: A Large-Scale Dataset for Robust Visual Table Understandingcited by 5
  9. 2025Beyond Relational: Semantic-Aware Multi-Modal Analytics with LLM-Native Query Optimizationcited by 5
  10. 2026Deep Tabular Research via Continual Experience-Driven Executioncited by 5
  11. 2025Bridging Language Models and Financial Analysiscited by 4
  12. 2025Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Modelscited by 4
  13. 2025TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognitioncited by 4
  14. 2026Infinity-Parser2 Technical Reportcited by 4
  15. 2025MathReal: We Keep It Real! A Real Scene Benchmark for Evaluating Math Reasoning in Multimodal Large Language Modelscited by 3
  16. 2025TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understandingcited by 3
  17. 2026Thinking with Tables: Enhancing Multi-Modal Tabular Understanding via Neuro-Symbolic Reasoningcited by 3
  18. 2026VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learningcited by 3
  19. 2025PILOT: A Promptable Interleaved Layout-aware OCR Transformercited by 2
  20. 2025Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligencecited by 2
  21. 2025Can MLLMs Absorb Math Reasoning Abilities from LLMs as Free Lunch?cited by 2
  22. 2025Lost in Translation and Noise: A Deep Dive into the Failure Modes of VLMs on Real-World Tablescited by 1
  23. 2026ST-Raptor: An Agentic System for Semi-Structured Table QAcited by 1
  24. 2026V-tableR1: Process-Supervised Multimodal Table Reasoning with Critic-Guided Policy Optimizationcited by 1
  25. 2025LLM4Hint: Leveraging Large Language Models for Hint Recommendation in Offline Query Optimizationcited by 0
  26. 2025A Multimodal Conversational Agent for Tabular Data Analysiscited by 0
  27. 2026Chart Deep Research in LVLMs via Parallel Relative Policy Optimizationcited by 0
  28. 2026HighlightBench: Benchmarking Markup-Driven Table Reasoning in Scientific Documentscited by 0
  29. 2026TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tablescited by 0
  30. 2026The Verbose Context Problem in Medical Recordscited by 0
  31. 2026Jina-OCR-v1: Efficient Document Parsing with Speculative Decoding and Dense Verifiable Rewardscited by 0
  32. 2026MERIT: Multimodal Evidence Reasoning over Image Tablescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MMTABREAL: Real-World Benchmark for Multimodal Table Understanding
  2. 2026Efficient Table Retrieval and Understanding with Multimodal Large Language Models
  3. 2025Does Table Source Matter? Benchmarking and Improving Multimodal Scientific Table Understanding and Reasoning
  4. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges
  5. 2026TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs
  6. 2024Large Language Model for Table Processing: A Survey
  7. 2024TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy
  8. 2024Knowledge-Aware Reasoning over Multimodal Semi-structured Tables