# TableRAG: A Retrieval Augmented Generation Framework for Heterogeneous Document Reasoning

arXiv: https://arxiv.org/abs/2506.10380 · HTML: https://mlsanity.com/paper/2506.10380/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)
- Retrieval-augmented generation / [Knowledge-graph and structured-knowledge RAG](https://mlsanity.com/rag/knowledge-graph-and-structured-knowledge-rag/)

## Builds on (most cited first, 25 of 25)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation](https://mlsanity.com/paper/2404.00610/) (2024)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [ReAcTable: Enhancing ReAct for Table Question Answering](https://mlsanity.com/paper/2310.00815/) (2023)
- [TableGPT2: A Large Multimodal Model with Tabular Data Integration](https://mlsanity.com/paper/2411.02059/) (2024)
- [Large Language Model for Table Processing: A Survey](https://mlsanity.com/paper/2402.05121/) (2024)
- [Tab-CoT: Zero-shot Tabular Chain of Thought](https://mlsanity.com/paper/2305.17812/) (2023)
- [TabSQLify: Enhancing Reasoning Capabilities of LLMs Through Table Decomposition](https://mlsanity.com/paper/2404.10150/) (2024)
- [The Chronicles of RAG: The Retriever, the Chunk and the Generator](https://mlsanity.com/paper/2401.07883/) (2024)
- [TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning](https://mlsanity.com/paper/2312.09039/) (2023)
- [REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models](https://mlsanity.com/paper/2402.07016/) (2024)
- [Can LLMs Master Math? Investigating Large Language Models on Math Stack Exchange](https://mlsanity.com/paper/2404.00344/) (2024)
- [H-STAR: LLM-driven Hybrid SQL-Text Adaptive Reasoning on Tables](https://mlsanity.com/paper/2407.05952/) (2024)
- [PyTond: Efficient Python Data Science on the Shoulders of Databases](https://mlsanity.com/paper/2407.11616/) (2024)

## Built on by (most cited first, 24 of 24)

- [LLM-Based Multi-Agent Blackboard System for Information Discovery in Data Science](https://mlsanity.com/paper/2510.01285/) (2025)
- [Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoning](https://mlsanity.com/paper/2509.02401/) (2025)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [Dingtalk DeepResearch: A Unified Multi Agent Framework for Adaptive Intelligence in Enterprise Environments](https://mlsanity.com/paper/2510.24760/) (2025)
- [PISA: A Pragmatic Psych-Inspired Unified Memory System for Enhanced AI Agency](https://mlsanity.com/paper/2510.15966/) (2025)
- [A Guide to Large Language Models in Modeling and Simulation: From Core Techniques to Critical Challenges](https://mlsanity.com/paper/2602.05883/) (2026)
- [An Agile Method for Implementing Retrieval Augmented Generation Tools in Industrial SMEs](https://mlsanity.com/paper/2508.21024/) (2025)
- [RELOOP: Recursive Retrieval with Multi-Hop Reasoner and Planners for Heterogeneous QA](https://mlsanity.com/paper/2510.20505/) (2025)
- [TabRAG: Improving Tabular Document Question Answering for Retrieval Augmented Generation via Structured Representations](https://mlsanity.com/paper/2511.06582/) (2025)
- [TDATR: Improving End-to-End Table Recognition via Table Detail-Aware Learning and Cell-Level Visual Alignment](https://mlsanity.com/paper/2603.22819/) (2026)
- [Overcoming the "Impracticality" of RAG: Proposing a Real-World Benchmark and Multi-Dimensional Diagnostic Framework](https://mlsanity.com/paper/2604.02640/) (2026)
- [Query Symbolically or Retrieve Semantically? A Dataset and Method for Semi-Structured Question Answering](https://mlsanity.com/paper/2605.27164/) (2026)
- [HVM-GraphRAG: Holistic-View Multimodal Graph Retrieval-Augmented Generation on Complex Document](https://mlsanity.com/paper/2607.24861/) (2026)
- [SQuARE: Structured Query & Adaptive Retrieval Engine For Tabular Formats](https://mlsanity.com/paper/2512.04292/) (2025)
- [Agentic DAG-Orchestrated Planner Framework for Multi-Modal, Multi-Hop Question Answering in Hybrid Data Lakes](https://mlsanity.com/paper/2603.14229/) (2026)
- [FT-RAG: A Fine-grained Retrieval-Augmented Generation Framework for Complex Table Reasoning](https://mlsanity.com/paper/2605.01495/) (2026)
- [Hypergraph Enterprise Agentic Reasoner over Heterogeneous Business Systems](https://mlsanity.com/paper/2605.14259/) (2026)
- [A Picture is Worth a Thousand Words? An Empirical Study of Aggregation Strategies for Visual Financial Document Retrieval](https://mlsanity.com/paper/2605.14581/) (2026)
- [ChartWalker: Benchmarking the Cross-Chart RAG Task with Hierarchical Knowledge Graphs](https://mlsanity.com/paper/2606.23997/) (2026)
- [Semantically Similar, Logically Distinct: Diagnosing the Semantic-Answerability Gap in Table RAG](https://mlsanity.com/paper/2607.17742/) (2026)
- [Hierarchical Reranking for Scalable Financial RAG System](https://mlsanity.com/paper/2607.27523/) (2026)
- [Think Inside the Chunk: RegulaRAG for Regulation-Compliant Scenario Generation using LLMs: A Case Study of UN Regulation No. 152](https://mlsanity.com/paper/2608.16394/) (2026)
- [VIGIL: Verifier-Informed Gated Improvement Loop for Spreadsheet Question Answering](https://mlsanity.com/paper/2610.04287/) (2026)
- [ProCARE: Real-World Study Automation via Profile-Grounded Evidence Contracts](https://mlsanity.com/paper/2610.16175/) (2026)

## Abstract (arXiv)

Retrieval-Augmented Generation (RAG) has demonstrated considerable effectiveness in open-domain question answering. However, when applied to heterogeneous documents, comprising both textual and tabular components, existing RAG approaches exhibit critical limitations. The prevailing practice of flattening tables and chunking strategies disrupts the intrinsic tabular structure, leads to information loss, and undermines the reasoning capabilities of LLMs in multi-hop, global queries. To address these challenges, we propose TableRAG, an SQL-based framework that unifies textual understanding and complex manipulations over tabular data. TableRAG iteratively operates in four steps: context-sensitive query decomposition, text retrieval, SQL programming and execution, and compositional intermediate answer generation. We also develop HeteQA, a novel benchmark designed to evaluate the multi-hop heterogeneous reasoning capabilities. Experimental results demonstrate that TableRAG consistently outperforms existing baselines on both public datasets and our HeteQA, establishing a new state-of-the-art for heterogeneous document question answering. We release TableRAG at https://github.com/yxh-y/TableRAG/tree/main.