# TAT-LLM: A Specialized Language Model for Discrete Reasoning over Tabular and Textual Data

arXiv: https://arxiv.org/abs/2401.13223 · HTML: https://mlsanity.com/paper/2401.13223/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 35 of its 52 references that are in the map.
- First posted on arXiv 24 Jan 2024.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct](https://mlsanity.com/paper/2308.09583/) (2023)
- [Instruction Tuning for Large Language Models: A Survey](https://mlsanity.com/paper/2308.10792/) (2023)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning](https://mlsanity.com/paper/2309.05653/) (2023)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data](https://mlsanity.com/paper/2004.07347/) (2020)
- [Specializing Smaller Language Models towards Multi-Step Reasoning](https://mlsanity.com/paper/2301.12726/) (2023)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [Open Question Answering over Tables and Text](https://mlsanity.com/paper/2010.10439/) (2020)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [Large Language Models are few(1)-shot Table Reasoners](https://mlsanity.com/paper/2210.06710/) (2022)
- [Towards Complex Document Understanding By Discrete Reasoning](https://mlsanity.com/paper/2207.11871/) (2022)
- [NumNet: Machine Reading Comprehension with Numerical Reasoning](https://mlsanity.com/paper/1910.06701/) (2019)
- [Tab-CoT: Zero-shot Tabular Chain of Thought](https://mlsanity.com/paper/2305.17812/) (2023)
- [Effective Distillation of Table-based Reasoning Ability from LLMs](https://mlsanity.com/paper/2309.13182/) (2023)
- [TSQA: Tabular Scenario Based Question Answering](https://mlsanity.com/paper/2101.11429/) (2021)
- [MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2309.04790/) (2023)
- [Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder](https://mlsanity.com/paper/2209.07692/) (2022)
- [ELASTIC: Numerical Reasoning with Adaptive Symbolic Compiler](https://mlsanity.com/paper/2210.10105/) (2022)
- [UniRPG: Unified Discrete Reasoning over Table and Text as Program Generation](https://mlsanity.com/paper/2210.08249/) (2022)

## Built on by (most cited first, 19 of 19)

- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios](https://mlsanity.com/paper/2403.19318/) (2024)
- [UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis](https://mlsanity.com/paper/2406.15187/) (2024)
- [M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework](https://mlsanity.com/paper/2411.06176/) (2024)
- [Text-Tuple-Table: Towards Information Integration in Text-to-Table Generation via Global Tuple Extraction](https://mlsanity.com/paper/2404.14215/) (2024)
- [Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Tasks](https://mlsanity.com/paper/2401.17585/) (2024)
- [Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answering](https://mlsanity.com/paper/2412.20145/) (2024)
- [Efficient Prompting for LLM-based Generative Internet of Things](https://mlsanity.com/paper/2406.10382/) (2024)
- [Enhancing Financial Question Answering with a Multi-Agent Reflection Framework](https://mlsanity.com/paper/2410.21741/) (2024)
- [ST-Raptor: LLM-Powered Semi-Structured Table Question Answering](https://mlsanity.com/paper/2508.18190/) (2025)
- [GoNoGo: An Efficient LLM-based Multi-Agent System for Streamlining Automotive Software Release Decision-Making](https://mlsanity.com/paper/2408.09785/) (2024)
- [Improved LLM Agents for Financial Document Question Answering](https://mlsanity.com/paper/2506.08726/) (2025)
- [DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics](https://mlsanity.com/paper/2402.11481/) (2024)
- [MATATA: Weakly Supervised End-to-End MAthematical Tool-Augmented Reasoning for Tabular Applications](https://mlsanity.com/paper/2411.18915/) (2024)
- [CF-RAG: A Dataset and Method for Carbon Footprint QA Using Retrieval-Augmented Generation](https://mlsanity.com/paper/2508.03489/) (2025)
- [DeALOG: Decentralized Multi-Agents Log-Mediated Reasoning Framework](https://mlsanity.com/paper/2602.00996/) (2026)
- [A Multi-Agent Approach for Claim Verification from Tabular Data Documents](https://mlsanity.com/paper/2604.17225/) (2026)
- [Reasoning4Sciences: Bridging Reasoning Language Models to All Scientific Branches](https://mlsanity.com/paper/2606.01145/) (2026)
- [CIPHER: Benchmarking Cross-record Inference over Privacy-Hardened Evidence Records](https://mlsanity.com/paper/2609.07022/) (2026)

## Abstract (arXiv)

In this work, we address question answering (QA) over a hybrid of tabular and textual data that are very common content on the Web (e.g. SEC filings), where discrete reasoning capabilities are often required. Recently, large language models (LLMs) like GPT-4 have demonstrated strong multi-step reasoning capabilities. We then consider harnessing the amazing power of LLMs to solve our task. We abstract a Step-wise Pipeline for tabular and textual QA, which consists of three key steps, including Extractor, Reasoner and Executor, and initially design an instruction to instantiate the pipeline and validate that GPT-4 outperforms all existing methods. However, utilizing an online LLM like GPT-4 holds various challenges in terms of cost, latency, and data security risk, which motivates us to specialize smaller LLMs in this task. We develop a TAT-LLM language model by fine-tuning LLaMA 2 with the training data generated automatically from existing expert-annotated datasets following the Step-wise Pipeline. The experimental results have verified that our TAT-LLM model can outperform all baseline models, including the previous best fine-tuned models and very large-scale LLMs like GPT-4 on FinQA, TAT-QA and TAT-DQA benchmarks.