# FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering arXiv: https://arxiv.org/abs/2404.18585 ยท HTML: https://mlsanity.com/paper/2404.18585/ ## Key facts - Cited by 16 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 29 references that are in the map. - First posted on arXiv 29 Apr 2024. ## Builds on (most cited first, 22 of 22) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019) - [Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task](https://mlsanity.com/paper/1809.08887/) (2018) - [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017) - [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015) - [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020) - [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021) - [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023) - [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021) - [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022) - [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023) - [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://mlsanity.com/paper/2203.00274/) (2022) - [ReAcTable: Enhancing ReAct for Table Question Answering](https://mlsanity.com/paper/2310.00815/) (2023) - [A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Models](https://mlsanity.com/paper/2210.12023/) (2022) - [RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations](https://mlsanity.com/paper/2306.14321/) (2023) - [Exploring the Numerical Reasoning Capabilities of Language Models: A Comprehensive Analysis on Tabular Data](https://mlsanity.com/paper/2311.02216/) (2023) - [Capturing Row and Column Semantics in Transformer Based Question Answering over Tables](https://mlsanity.com/paper/2104.08303/) (2021) - [S3HQA: A Three-Stage Approach for Multi-hop Text-Table Hybrid Question Answering](https://mlsanity.com/paper/2305.11725/) (2023) - [TabLeX: A Benchmark Dataset for Structure and Content Information Extraction from Scientific Tables](https://mlsanity.com/paper/2105.06400/) (2021) - [Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning](https://mlsanity.com/paper/2108.00578/) (2021) - [AssistSR: Task-oriented Video Segment Retrieval for Personal AI Assistant](https://mlsanity.com/paper/2111.15050/) (2021) ## Built on by (most cited first, 16 of 16) - [SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation](https://mlsanity.com/paper/2406.14991/) (2024) - [Can LLM Graph Reasoning Generalize beyond Pattern Memorization?](https://mlsanity.com/paper/2406.15992/) (2024) - [Efficient Multi-Agent Collaboration with Tool Use for Online Planning in Complex Table Question Answering](https://mlsanity.com/paper/2412.20145/) (2024) - [Tabular Data Understanding with LLMs: A Survey of Recent Advances and Challenges](https://mlsanity.com/paper/2508.00217/) (2025) - [Exploring the Robustness of Language Models for Tabular Question Answering via Attention Analysis](https://mlsanity.com/paper/2406.12719/) (2024) - [NeedleInATable: Exploring Long-Context Capability of Large Language Models towards Long-Structured Tables](https://mlsanity.com/paper/2504.06560/) (2025) - [Texts or Images? A Fine-grained Analysis on the Effectiveness of Input Representations and Models for Table Question Answering](https://mlsanity.com/paper/2505.14131/) (2025) - [Table Question Answering in the Era of Large Language Models: A Comprehensive Survey of Tasks, Methods, and Evaluation](https://mlsanity.com/paper/2510.09671/) (2025) - [TableDreamer: Progressive and Weakness-guided Data Synthesis from Scratch for Table Instruction Tuning](https://mlsanity.com/paper/2506.08646/) (2025) - [RoToR: Towards More Reliable Responses for Order-Invariant Inputs](https://mlsanity.com/paper/2502.08662/) (2025) - [Structural Deep Encoding for Table Question Answering](https://mlsanity.com/paper/2503.01457/) (2025) - [p2-TQA: A Process-based Preference Learning Framework for Self-Improving Table Question Answering Models](https://mlsanity.com/paper/2505.17565/) (2025) - [How to Utilize Complementary Vision-Text Information for 2D Structure Understanding](https://mlsanity.com/paper/2603.16245/) (2026) - [Towards Robust Numerical Claim Verification](https://mlsanity.com/paper/2610.00689/) (2026) - [Understanding Errors in LLM-Based Question Answering over Imperfect Tables](https://mlsanity.com/paper/2610.04687/) (2026) - [Typed Decision Models on Tabular Data](https://mlsanity.com/paper/2610.19536/) (2026) ## Abstract (arXiv) Table Question Answering (TQA) aims at composing an answer to a question based on tabular data. While prior research has shown that TQA models lack robustness, understanding the underlying cause and nature of this issue remains predominantly unclear, posing a significant obstacle to the development of robust TQA systems. In this paper, we formalize three major desiderata for a fine-grained evaluation of robustness of TQA systems. They should (i) answer questions regardless of alterations in table structure, (ii) base their responses on the content of relevant cells rather than on biases, and (iii) demonstrate robust numerical reasoning capabilities. To investigate these aspects, we create and publish a novel TQA evaluation benchmark in English. Our extensive experimental analysis reveals that none of the examined state-of-the-art TQA systems consistently excels in these three aspects. Our benchmark is a crucial instrument for monitoring the behavior of TQA systems and paves the way for the development of robust TQA systems. We release our benchmark publicly.