# Table-To-Text generation and pre-training with TabT5

arXiv: https://arxiv.org/abs/2210.09162 · HTML: https://mlsanity.com/paper/2210.09162/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [FinQA: A Dataset of Numerical Reasoning over Financial Data](https://mlsanity.com/paper/2109.00122/) (2021)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [TABBIE: Pretrained Representations of Tabular Data](https://mlsanity.com/paper/2105.02584/) (2021)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [TableFormer: Robust Transformer Modeling for Table-Text Encoding](https://mlsanity.com/paper/2203.00274/) (2022)
- [MATE: Multi-view Attention for Table Transformer Efficiency](https://mlsanity.com/paper/2109.04312/) (2021)
- [SpreadsheetCoder: Formula Prediction from Semi-structured Context](https://mlsanity.com/paper/2106.15339/) (2021)
- [Answering Conversational Questions on Structured Data without Logical Forms](https://mlsanity.com/paper/1908.11787/) (2019)

## Built on by (most cited first, 20 of 20)

- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [DePlot: One-shot visual language reasoning by plot-to-table translation](https://mlsanity.com/paper/2212.10505/) (2022)
- [Generative Table Pre-training Empowers Models for Tabular Prediction](https://mlsanity.com/paper/2305.09696/) (2023)
- [Transforming Science with Large Language Models: A Survey on AI-assisted Scientific Discovery, Experimentation, Content Generation, and Evaluation](https://mlsanity.com/paper/2502.05151/) (2025)
- [HYTREL: Hypergraph-enhanced Tabular Data Representation Learning](https://mlsanity.com/paper/2307.08623/) (2023)
- [MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering](https://mlsanity.com/paper/2305.12820/) (2023)
- [Unifying Structured Data as Graph for Data-to-Text Pre-Training](https://mlsanity.com/paper/2401.01183/) (2024)
- [DOMINO: A Dual-System for Multi-step Visual Language Reasoning](https://mlsanity.com/paper/2310.02804/) (2023)
- [Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks](https://mlsanity.com/paper/2310.00789/) (2023)
- [Structsum Generation for Faster Text Comprehension](https://mlsanity.com/paper/2401.06837/) (2024)
- [Generating Realistic Tabular Data with Large Language Models](https://mlsanity.com/paper/2410.21717/) (2024)
- [APOLLO: An Optimized Training Approach for Long-form Numerical Reasoning](https://mlsanity.com/paper/2212.07249/) (2022)
- [Organizing Unstructured Image Collections using Natural Language](https://mlsanity.com/paper/2410.05217/) (2024)
- [Faithful Chart Summarization with ChaTS-Pi](https://mlsanity.com/paper/2405.19094/) (2024)
- [Retrieval-Augmented Chain-of-Thought in Semi-structured Domains](https://mlsanity.com/paper/2310.14435/) (2023)
- [Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations](https://mlsanity.com/paper/2212.08780/) (2022)
- [ReTAG: Reasoning Aware Table to Analytic Text Generation](https://mlsanity.com/paper/2305.11826/) (2023)
- [BenchPress: A Human-in-the-Loop Annotation System for Rapid Text-to-SQL Benchmark Curation](https://mlsanity.com/paper/2510.13853/) (2025)
- [Few-Shot Data-to-Text Generation via Unified Representation and Multi-Source Learning](https://mlsanity.com/paper/2308.05317/) (2023)
- [FreshTab: Sourcing Fresh Data for Table-to-Text Generation Evaluation](https://mlsanity.com/paper/2510.13598/) (2025)

## Abstract (arXiv)

Encoder-only transformer models have been successfully applied to different table understanding tasks, as in TAPAS (Herzig et al., 2020). A major limitation of these architectures is that they are constrained to classification-like tasks such as cell selection or entailment detection. We present TABT5, an encoder-decoder model that generates natural language text based on tables and textual inputs. TABT5 overcomes the encoder-only limitation by incorporating a decoder component and leverages the input structure with table specific embeddings and pre-training. TABT5 achieves new state-of-the-art results on several domains, including spreadsheet formula prediction with a 15% increase in sequence accuracy, QA with a 2.5% increase in sequence accuracy and data-to-text generation with a 2.5% increase in BLEU.