# RealHiTBench: A Comprehensive Realistic Hierarchical Table Benchmark for Evaluating LLM-Based Table Analysis

arXiv: https://arxiv.org/abs/2506.13405 · HTML: https://mlsanity.com/paper/2506.13405/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 45 references that are in the map.
- First posted on arXiv 16 Jun 2025.

## Builds on (most cited first, 30 of 32)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Bias](https://mlsanity.com/paper/2306.15895/) (2023)
- [FeTaQA: Free-form Table Question Answering](https://mlsanity.com/paper/2104.00369/) (2021)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [TableLlama: Towards Open Large Generalist Models for Tables](https://mlsanity.com/paper/2311.09206/) (2023)
- [TableBench: A Comprehensive and Complex Benchmark for Table Question Answering](https://mlsanity.com/paper/2408.09174/) (2024)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation](https://mlsanity.com/paper/2108.06712/) (2021)
- [SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation](https://mlsanity.com/paper/2406.14991/) (2024)
- [TableGPT2: A Large Multimodal Model with Tabular Data Integration](https://mlsanity.com/paper/2411.02059/) (2024)
- [TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domains](https://mlsanity.com/paper/2404.19205/) (2024)
- [SCITAB: A Challenging Benchmark for Compositional Reasoning and Claim Verification on Scientific Tables](https://mlsanity.com/paper/2305.13186/) (2023)
- [Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs](https://mlsanity.com/paper/2310.10358/) (2023)
- [AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industry](https://mlsanity.com/paper/2106.12944/) (2021)
- [Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queries](https://mlsanity.com/paper/2312.13671/) (2023)
- [Large Language Models are Complex Table Parsers](https://mlsanity.com/paper/2312.11521/) (2023)
- [Answering Conversational Questions on Structured Data without Logical Forms](https://mlsanity.com/paper/1908.11787/) (2019)
- [Effective Distillation of Table-based Reasoning Ability from LLMs](https://mlsanity.com/paper/2309.13182/) (2023)
- [GraphOTTER: Evolving LLM-based Graph Reasoning for Complex Table Question Answering](https://mlsanity.com/paper/2412.01230/) (2024)
- [Interpretable LLM-based Table Question Answering](https://mlsanity.com/paper/2412.12386/) (2024)
- [GPT is Not an Annotator: The Necessity of Human Annotation in Fairness Benchmark Construction](https://mlsanity.com/paper/2405.15760/) (2024)
- [Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks](https://mlsanity.com/paper/2310.00789/) (2023)
- [Enhancing Table Recognition with Vision LLMs: A Benchmark and Neighbor-Guided Toolchain Reasoner](https://mlsanity.com/paper/2412.20662/) (2024)

## Built on by (most cited first, 12 of 12)

- [Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows](https://mlsanity.com/paper/2512.13168/) (2025)
- [FAITH: A Framework for Assessing Intrinsic Tabular Hallucinations in Finance](https://mlsanity.com/paper/2508.05201/) (2025)
- [SheetBrain: A Neuro-Symbolic Agent for Accurate Reasoning over Complex and Large Spreadsheets](https://mlsanity.com/paper/2510.19247/) (2025)
- [TableGPT-R1: Advancing Tabular Reasoning Through Reinforcement Learning](https://mlsanity.com/paper/2512.20312/) (2025)
- [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025)
- [Deep Tabular Research via Continual Experience-Driven Execution](https://mlsanity.com/paper/2603.09151/) (2026)
- [WildTableBench: Benchmarking Multimodal Foundation Models on Table Understanding In the Wild](https://mlsanity.com/paper/2605.01018/) (2026)
- [TABVERSE: Benchmarking Cross-Format Table Understanding in LLMs and VLMs](https://mlsanity.com/paper/2606.09578/) (2026)
- [An Empirical Investigation of Robustness in Large Language Models under Tabular Distortions](https://mlsanity.com/paper/2601.05009/) (2026)
- [TableVision: A Large-Scale Benchmark for Spatially Grounded Reasoning over Complex Hierarchical Tables](https://mlsanity.com/paper/2604.03660/) (2026)
- [CITBench: A Comprehensive Benchmark for Interactive Tabular Data Processing with LLMs](https://mlsanity.com/paper/2608.00018/) (2026)
- [Privacy-Preserving Data Drift Detection and Recovery for Large-Scale LLM Applications via Proxy Representations](https://mlsanity.com/paper/2608.08245/) (2026)

## Abstract (arXiv)

With the rapid advancement of Large Language Models (LLMs), there is an increasing need for challenging benchmarks to evaluate their capabilities in handling complex tabular data. However, existing benchmarks are either based on outdated data setups or focus solely on simple, flat table structures. In this paper, we introduce RealHiTBench, a comprehensive benchmark designed to evaluate the performance of both LLMs and Multimodal LLMs (MLLMs) across a variety of input formats for complex tabular data, including LaTeX, HTML, and PNG. RealHiTBench also includes a diverse collection of tables with intricate structures, spanning a wide range of task types. Our experimental results, using 25 state-of-the-art LLMs, demonstrate that RealHiTBench is indeed a challenging benchmark. Moreover, we also develop TreeThinker, a tree-based pipeline that organizes hierarchical headers into a tree structure for enhanced tabular reasoning, validating the importance of improving LLMs' perception of table hierarchies. We hope that our work will inspire further research on tabular data reasoning and the development of more robust models. The code and data are available at https://github.com/cspzyy/RealHiTBench.