# TableFormer: Robust Transformer Modeling for Table-Text Encoding

arXiv: https://arxiv.org/abs/2203.00274 · HTML: https://mlsanity.com/paper/2203.00274/

## Key facts

- Cited by 91 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 21 references that are in the map.
- First posted on arXiv 1 Mar 2022.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Self-Attention with Relative Position Representations](https://mlsanity.com/paper/1803.02155/) (2018)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers](https://mlsanity.com/paper/1911.04942/) (2019)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Do Transformers Really Perform Bad for Graph Representation?](https://mlsanity.com/paper/2106.05234/) (2021)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing](https://mlsanity.com/paper/2009.13845/) (2020)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [Understanding tables with intermediate pre-training](https://mlsanity.com/paper/2010.00571/) (2020)
- [MATE: Multi-view Attention for Table Transformer Efficiency](https://mlsanity.com/paper/2109.04312/) (2021)
- [Answering Conversational Questions on Structured Data without Logical Forms](https://mlsanity.com/paper/1908.11787/) (2019)

## Built on by (most cited first, 30 of 91)

- [Unified Training of Universal Time Series Forecasting Transformers](https://mlsanity.com/paper/2402.02592/) (2024)
- [Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond](https://mlsanity.com/paper/2304.13712/) (2023)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [TransTab: Learning Transferable Tabular Transformers Across Tables](https://mlsanity.com/paper/2205.09328/) (2022)
- [MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data](https://mlsanity.com/paper/2206.01347/) (2022)
- [Shortcut Learning of Large Language Models in Natural Language Understanding](https://mlsanity.com/paper/2208.11857/) (2022)
- [TableLLM: Enabling Tabular Data Manipulation by LLMs in Real Office Usage Scenarios](https://mlsanity.com/paper/2403.19318/) (2024)
- [TableGPT2: A Large Multimodal Model with Tabular Data Integration](https://mlsanity.com/paper/2411.02059/) (2024)
- [A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions](https://mlsanity.com/paper/2208.13629/) (2022)
- [RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations](https://mlsanity.com/paper/2306.14321/) (2023)
- [A Survey on Table Question Answering: Recent Advances](https://mlsanity.com/paper/2207.05270/) (2022)
- [SeqZero: Few-shot Compositional Semantic Parsing with Sequential Prompts and Zero-shot Models](https://mlsanity.com/paper/2205.07381/) (2022)
- [HYTREL: Hypergraph-enhanced Tabular Data Representation Learning](https://mlsanity.com/paper/2307.08623/) (2023)
- [TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data](https://mlsanity.com/paper/2407.07582/) (2024)
- [ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples](https://mlsanity.com/paper/2210.12374/) (2022)
- [Large Language Models are Complex Table Parsers](https://mlsanity.com/paper/2312.11521/) (2023)
- [Active Prompt Learning in Vision Language Models](https://mlsanity.com/paper/2311.11178/) (2023)
- [CABINET: Content Relevance based Noise Reduction for Table Question Answering](https://mlsanity.com/paper/2402.01155/) (2024)
- [ProTrix: Building Models for Planning and Reasoning over Tables with Sentence Context](https://mlsanity.com/paper/2403.02177/) (2024)
- [Table-To-Text generation and pre-training with TabT5](https://mlsanity.com/paper/2210.09162/) (2022)
- [Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolution](https://mlsanity.com/paper/2408.10548/) (2024)
- [Tele-Knowledge Pre-training for Fault Analysis](https://mlsanity.com/paper/2210.11298/) (2022)
- [SUBS: Subtree Substitution for Compositional Semantic Parsing](https://mlsanity.com/paper/2205.01538/) (2022)
- [FREB-TQA: A Fine-Grained Robustness Evaluation Benchmark for Table Question Answering](https://mlsanity.com/paper/2404.18585/) (2024)
- [LLM/Agent-as-Data-Analyst: A Survey](https://mlsanity.com/paper/2509.23988/) (2025)
- [NameGuess: Column Name Expansion for Tabular Data](https://mlsanity.com/paper/2310.13196/) (2023)
- [Unlock the Potential of Large Language Models for Predictive Tabular Tasks in Data Science with Table-Specific Pretraining](https://mlsanity.com/paper/2403.20208/) (2024)
- [Table Retrieval May Not Necessitate Table-specific Model Design](https://mlsanity.com/paper/2205.09843/) (2022)
- [Designing Robust Transformers using Robust Kernel Density Estimation](https://mlsanity.com/paper/2210.05794/) (2022)
- [cPAPERS: A Dataset of Situated and Multimodal Interactive Conversations in Scientific Papers](https://mlsanity.com/paper/2406.08398/) (2024)

## Abstract (arXiv)

Understanding tables is an important aspect of natural language understanding. Existing models for table understanding require linearization of the table structure, where row or column order is encoded as an unwanted bias. Such spurious biases make the model vulnerable to row and column order perturbations. Additionally, prior work has not thoroughly modeled the table structures or table-text alignments, hindering the table-text understanding ability. In this work, we propose a robust and structurally aware table-text encoding architecture TableFormer, where tabular structural biases are incorporated completely through learnable attention biases. TableFormer is (1) strictly invariant to row and column orders, and, (2) could understand tables better due to its tabular inductive biases. Our evaluations showed that TableFormer outperforms strong baselines in all settings on SQA, WTQ and TabFact table reasoning datasets, and achieves state-of-the-art performance on SQA, especially when facing answer-invariant row and column order perturbations (6% improvement over the best baseline), because previous SOTA models' performance drops by 4% - 6% when facing such perturbations while TableFormer is not affected.