# Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs

arXiv: https://arxiv.org/abs/2402.12424 · HTML: https://mlsanity.com/paper/2402.12424/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 20 of its 49 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [Reasoning with Language Model Prompting: A Survey](https://mlsanity.com/paper/2212.09597/) (2022)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [A Comprehensive Exploration on WikiSQL with Table-Aware Word Contextualization](https://mlsanity.com/paper/1902.01069/) (2019)
- [Lemur: Harmonizing Natural Language and Code for Language Agents](https://mlsanity.com/paper/2310.06830/) (2023)
- [TableGPT: Towards Unifying Tables, Nature Language and Commands into One GPT](https://mlsanity.com/paper/2307.08674/) (2023)
- [An In-depth Look at Gemini's Language Abilities](https://mlsanity.com/paper/2312.11444/) (2023)
- [Observatory: Characterizing Embeddings of Relational Tables](https://mlsanity.com/paper/2310.07736/) (2023)

## Built on by (most cited first, 30 of 40)

- [From Automation to Autonomy: A Survey on Large Language Models in Scientific Discovery](https://mlsanity.com/paper/2505.13259/) (2025)
- [A Survey on Knowledge-Oriented Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.10677/) (2025)
- [Table-R1: Inference-Time Scaling for Table Reasoning](https://mlsanity.com/paper/2505.23621/) (2025)
- [SciVer: Evaluating Foundation Models for Multimodal Scientific Claim Verification](https://mlsanity.com/paper/2506.15569/) (2025)
- [TableEval: A Real-World Benchmark for Complex, Multilingual, and Multi-Structured Table Question Answering](https://mlsanity.com/paper/2506.03949/) (2025)
- [Multimodal Tabular Reasoning with Privileged Structured Information](https://mlsanity.com/paper/2506.04088/) (2025)
- [ST-Raptor: LLM-Powered Semi-Structured Table Question Answering](https://mlsanity.com/paper/2508.18190/) (2025)
- [A Survey of AI Scientists](https://mlsanity.com/paper/2510.23045/) (2025)
- [OWL: Geometry-Aware Spatial Reasoning for Audio Large Language Models](https://mlsanity.com/paper/2509.26140/) (2025)
- [Aurelia: Test-time Reasoning Distillation in Audio-Visual LLMs](https://mlsanity.com/paper/2503.23219/) (2025)
- [MTabVQA: Evaluating Multi-Tabular Reasoning of Language Models in Visual Space](https://mlsanity.com/paper/2506.11684/) (2025)
- [HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization](https://mlsanity.com/paper/2502.17315/) (2025)
- [Texts or Images? A Fine-grained Analysis on the Effectiveness of Input Representations and Models for Table Question Answering](https://mlsanity.com/paper/2505.14131/) (2025)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [Vision Language Models for Spreadsheet Understanding: Challenges and Opportunities](https://mlsanity.com/paper/2405.16234/) (2024)
- [ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art](https://mlsanity.com/paper/2410.01733/) (2024)
- [TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning](https://mlsanity.com/paper/2506.08646/) (2025)
- [Reasoning Factual Knowledge in Structured Data with Large Language Models](https://mlsanity.com/paper/2408.12188/) (2024)
- [Visual Language Models show widespread visual deficits on neuropsychological tests](https://mlsanity.com/paper/2504.10786/) (2025)
- [TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models](https://mlsanity.com/paper/2506.18421/) (2025)
- [TableDART: Dynamic Adaptive Multi-Modal Routing for Table Understanding](https://mlsanity.com/paper/2509.14671/) (2025)
- [Toward Real-World Table Agents: Capabilities, Workflows, and Design Principles for LLM-based Table Intelligence](https://mlsanity.com/paper/2507.10281/) (2025)
- [AraTable: Benchmarking LLMs' Reasoning and Understanding of Arabic Tabular Data](https://mlsanity.com/paper/2507.18442/) (2025)
- [Mind the (Language) Gap: Towards Probing Numerical and Cross-Lingual Limits of LVLMs](https://mlsanity.com/paper/2508.17334/) (2025)
- [CFMS: A Coarse-to-Fine Multimodal Synthesis Framework for Enhanced Tabular Reasoning](https://mlsanity.com/paper/2604.10973/) (2026)
- [Talking with Tables for Better LLM Factual Data Interactions](https://mlsanity.com/paper/2412.17189/) (2024)
- [GeoGrid-Bench: Can Foundation Models Understand Multimodal Gridded Geo-Spatial Data?](https://mlsanity.com/paper/2505.10714/) (2025)
- [The Power of Order: Fooling LLMs with Adversarial Table Permutations](https://mlsanity.com/paper/2605.00445/) (2026)
- [TableVista: Benchmarking Multimodal Table Reasoning under Visual and Structural Complexity](https://mlsanity.com/paper/2605.05955/) (2026)

## Abstract (arXiv)

In this paper, we investigate the effectiveness of various LLMs in interpreting tabular data through different prompting strategies and data formats. Our analyses extend across six benchmarks for table-related tasks such as question-answering and fact-checking. We introduce for the first time the assessment of LLMs' performance on image-based table representations. Specifically, we compare five text-based and three image-based table representations, demonstrating the role of representation and prompting on LLM performance. Our study provides insights into the effective use of LLMs on table-related tasks.