mlsanityData to 14 Oct 2026

A Closer Look into LLMs for Table Understanding

arXiv 2603.15402 ↗First posted 16 Mar 2026cs.CL cs.AIcited by 0

On the map: Mixture-of-experts → MoE large language models

Abstract (arXiv)

Despite the success of Large Language Models (LLMs) in table understanding, their internal mechanisms remain unclear. In this paper, we conduct an empirical study on 16 LLMs, covering general LLMs, specialist tabular LLMs, and Mixture-of-Experts (MoE) models, to explore how LLMs understand tabular data and perform downstream tasks. Our analysis focus on 4 dimensions including the attention dynamics, the effective layer depth, the expert activation, and the impacts of input designs. Key findings include: (1) LLMs follow a three-phase attention pattern -- early layers scan the table broadly, middle layers localize relevant cells, and late layers amplify their contributions; (2) tabular tasks require deeper layers than math reasoning to reach stable predictions; (3) MoE models activate table-specific experts in middle layers, with early and late layers sharing general-purpose experts; (4) Chain-of-Thought prompting increases table attention, further enhanced by table-tuning. We hope these findings and insights can facilitate interpretability and future research on table-related tasks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 36 of its 52 references that are in the map. First posted on arXiv 16 Mar 2026.

Builds on

36 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free10 May
  2. 2024A Closer Look into Mixture-of-Experts in Large Language Models26 Jun

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
  6. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 428
  7. 2023StructGPT: A General Framework for Large Language Model to Reason over Structured Datacited by 343
  8. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 311
  9. 2024DuoAttention: Efficient Long-Context LLM Inference with Retrieval and Streaming Headscited by 288
  10. 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 243
  11. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 232
  12. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
22 more
  1. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  2. 2023Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoningcited by 178
  3. 2023TableLlama: Towards Open Large Generalist Models for Tablescited by 137
  4. 2021HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generationcited by 118
  5. 2022Large Language Models are few(1)-shot Table Reasonerscited by 116
  6. 2024TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarioscited by 77
  7. 2024Large Language Model for Table Processing: A Surveycited by 52
  8. 2025Attention Sinks and Compression Valleys in LLMs are Two Sides of the Same Coincited by 52
  9. 2025Do Language Models Use Their Depth Efficiently?cited by 50
  10. 2023Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMscited by 49
  11. 2021AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industrycited by 44
  12. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMscited by 40
  13. 2025Table-R1: Region-based Reinforcement Learning for Table Understandingcited by 37
  14. 2023Analyzing Feed-Forward Blocks in Transformers through the Lens of Attention Mapscited by 33
  15. 2025Multilingual Routing in Mixture-of-Expertscited by 29
  16. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understandingcited by 20
  17. 2023An Empirical Study of Instruction-tuning Large Language Models in Chinesecited by 17
  18. 2025Understanding and Leveraging the Expert Specialization of Context Faithfulness in Mixture-of-Experts LLMscited by 12
  19. 2025Rethinking Table Instruction Tuningcited by 10
  20. 2025Can GRPO Boost Complex Multimodal Table Understanding?cited by 10
  21. 2024Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysiscited by 9
  22. 2025p2-TQA: A Process-based Preference Learning Framework for Self-Improving Table Question Answering Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges
  2. 2025How well do LLMs reason over tabular data, really?
  3. 2023Rethinking Tabular Data Understanding with Large Language Models
  4. 2025Utilizing Training Data to Improve LLM Reasoning for Tabular Understanding
  5. 2025Table Understanding and (Multimodal) LLMs: A Cross-Domain Case Study on Scientific vs. Non-Scientific Data
  6. 2024A Survey of Table Reasoning with Large Language Models
  7. 2023Effective Distillation of Table-based Reasoning Ability from LLMs
  8. 2026TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs