# Uncovering Limitations of Large Language Models in Information Seeking from Tables

arXiv: https://arxiv.org/abs/2406.04113 · HTML: https://mlsanity.com/paper/2406.04113/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 6 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [Evaluating Large Language Models at Evaluating Instruction Following](https://mlsanity.com/paper/2310.07641/) (2023)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [FeTaQA: Free-form Table Question Answering](https://mlsanity.com/paper/2104.00369/) (2021)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019)
- [Logical Natural Language Generation from Open-Domain Tables](https://mlsanity.com/paper/2004.10404/) (2020)
- [Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data](https://mlsanity.com/paper/2311.02216/) (2023)
- [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://mlsanity.com/paper/2205.03972/) (2022)
- [Effective Distillation of Table-based Reasoning Ability from LLMs](https://mlsanity.com/paper/2309.13182/) (2023)

## Built on by (most cited first, 9 of 9)

- [FCMR: Robust Evaluation of Financial Cross-Modal Multi-Hop Reasoning](https://mlsanity.com/paper/2412.12567/) (2024)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [Seek and Solve Reasoning for Table Question Answering](https://mlsanity.com/paper/2409.05286/) (2024)
- [Piece of Table: A Divide-and-Conquer Approach for Selecting Subtables in Table Question Answering](https://mlsanity.com/paper/2412.07629/) (2024)
- [INVARLLM: LLM-assisted Physical Invariant Extraction for Cyber-Physical Systems Anomaly Detection](https://mlsanity.com/paper/2411.10918/) (2024)
- [Document-Level Tabular Numerical Cross-Checking: A Coarse-to-Fine Approach](https://mlsanity.com/paper/2506.13328/) (2025)
- [Navigating Large-Scale Document Collections: MuDABench for Multi-Document Analytical QA](https://mlsanity.com/paper/2604.22239/) (2026)
- [TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Action](https://mlsanity.com/paper/2509.15098/) (2025)
- [TableNet A Large-Scale Table Dataset with LLM-Powered Autonomous](https://mlsanity.com/paper/2604.13041/) (2026)

## Abstract (arXiv)

Tables are recognized for their high information density and widespread usage, serving as essential sources of information. Seeking information from tables (TIS) is a crucial capability for Large Language Models (LLMs), serving as the foundation of knowledge-based Q&A systems. However, this field presently suffers from an absence of thorough and reliable evaluation. This paper introduces a more reliable benchmark for Table Information Seeking (TabIS). To avoid the unreliable evaluation caused by text similarity-based metrics, TabIS adopts a single-choice question format (with two options per question) instead of a text generation format. We establish an effective pipeline for generating options, ensuring their difficulty and quality. Experiments conducted on 12 LLMs reveal that while the performance of GPT-4-turbo is marginally satisfactory, both other proprietary and open-source models perform inadequately. Further analysis shows that LLMs exhibit a poor understanding of table structures, and struggle to balance between TIS performance and robustness against pseudo-relevant tables (common in retrieval-augmented systems). These findings uncover the limitations and potential challenges of LLMs in seeking information from tables. We release our data and code to facilitate further research in this field.