mlsanityData to 7 Oct 2026

TReB: A Comprehensive Benchmark for Evaluating Table Reasoning Capabilities of Large Language Models

arXiv 2506.18421 ↗First posted 23 Jun 2025cs.CL cs.AIcited by 3

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

The majority of data in businesses and industries is stored in tables, databases, and data warehouses. Reasoning with table-structured data poses significant challenges for large language models (LLMs) due to its hidden semantics, inherent complexity, and structured nature. One of these challenges is lacking an effective evaluation benchmark fairly reflecting the performances of LLMs on broad table reasoning abilities. In this paper, we fill in this gap by presenting a comprehensive table reasoning benchmark, TReB. Firstly, we propose a taxonomy to systematically measure both shallow table understanding abilities and deep table reasoning abilities, covering a total of 26 sub-tasks. We then construct a high quality dataset through a dedicated data processing and synthesis procedure. Based on these well-constructed samples, we design an evaluation framework to robustly measure table reasoning capabilities with three distinct inference modes. Experimental results with our data and framework reveal that existing LLMs still have significant room for improvement in addressing the complex and real world table related tasks. Both the dataset and evaluation framework are publicly available, with the dataset hosted on https://huggingface.co/datasets/JT-LM/JIUTIAN-TReB, and the framework on https://github.com/JT-LM/jiutian-treb.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 56 of its 65 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

56 of 65 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Table-R1: Inference-Time Scaling for Table Reasoning29 May
  2. 2024Tables as Texts or Images: Evaluating the Table Reasoning Ability of LLMs and MLLMs19 Feb
  3. 2024A Survey of Table Reasoning with Large Language Models13 Feb

Other references · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  8. 2025Qwen3 Technical Reportcited by 8,195
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  10. 2024Qwen2.5 Technical Reportcited by 4,713
  11. 2023Mistral 7Bcited by 3,065
  12. 2024Qwen2 Technical Reportcited by 2,076
38 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2024A Survey on LLM-as-a-Judgecited by 1,384
  3. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  4. 2024Qwen2.5-Coder Technical Reportcited by 1,151
  5. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  6. 2022Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Taskscited by 1,057
  7. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  8. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  9. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  10. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 728
  11. 2024Yi: Open Foundation Models by 01.AIcited by 706
  12. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  13. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  14. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  15. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  16. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  17. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
  18. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 267
  19. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  20. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 263
  21. 2023Magicoder: Empowering Code Generation with OSS-Instructcited by 251
  22. 2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
  23. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  24. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  25. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 173
  26. 2023MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQLcited by 158
  27. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  28. 2024TableBench: A Comprehensive and Complex Benchmark for Table Question Answeringcited by 127
  29. 2023Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Studycited by 116
  30. 2024MathBench: Evaluating the Theory and Application Proficiency of LLMs with a Hierarchical Mathematics Benchmarkcited by 103
  31. 2023XTab: Cross-table Pretraining for Tabular Transformerscited by 81
  32. 2024TableGPT2: A Large Multimodal Model with Tabular Data Integrationcited by 69
  33. 2024TableVQA-Bench: A Visual Question Answering Benchmark on Multiple Table Domainscited by 68
  34. 2024Large Language Model for Table Processing: A Surveycited by 52
  35. 2021AIT-QA: Question Answering Dataset over Complex Tables in the Airline Industrycited by 44
  36. 2024McEval: Massively Multilingual Code Evaluationcited by 42
  37. 2023UHGEval: Benchmarking the Hallucination of Chinese Large Language Models via Unconstrained Generationcited by 22
  38. 2024TQA-Bench: Evaluating LLMs for Multi-Table Question Answeringcited by 17

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
  2. 2025JT-DA: Enhancing Data Analysis with Tool-Integrated Table Reasoning Large Language Modelscited by 0
  3. 2026ProfiliTable: Profiling-Driven Tabular Data Processing via Agentic Workflowscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding
  2. 2026Probing How Scalable Table Data Enhances General Long-Context Reasoning
  3. 2023Rethinking Tabular Data Understanding with Large Language Models
  4. 2023TAP4LLM: Table Provider on Sampling, Augmenting, and Packing Semi-structured Data for Large Language Model Reasoning
  5. 2025RUST-BENCH: Benchmarking LLM Reasoning on Unstructured Text within Structured Tables
  6. 2026ProgramTab: Boosting Table Reasoning of LLMs via Programmatic Paradigm
  7. 2022Large Language Models are few(1)-shot Table Reasoners
  8. 2024OpenTab: Advancing Large Language Models as Open-domain Table Reasoners