# PixT3: Pixel-based Table-To-Text Generation

arXiv: https://arxiv.org/abs/2311.09808 · HTML: https://mlsanity.com/paper/2311.09808/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 60 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [LayoutLMv3: Pre-training for Document AI with Unified Text and Image Masking](https://mlsanity.com/paper/2204.08387/) (2022)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding](https://mlsanity.com/paper/2306.17107/) (2023)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019)
- [Text-to-Text Pre-Training for Data-to-Text Tasks](https://mlsanity.com/paper/2005.10433/) (2020)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [Logical Natural Language Generation from Open-Domain Tables](https://mlsanity.com/paper/2004.10404/) (2020)
- [End-to-end Document Recognition and Understanding with Dessurt](https://mlsanity.com/paper/2203.16618/) (2022)
- [Learning Neural Templates for Text Generation](https://mlsanity.com/paper/1808.10122/) (2018)
- [HTLM: Hyper-Text Pre-Training and Prompting of Language Models](https://mlsanity.com/paper/2107.06955/) (2021)
- [Plan-then-Generate: Controlled Data-to-Text Generation via Planning](https://mlsanity.com/paper/2108.13740/) (2021)
- [Data-to-text Generation with Entity Modeling](https://mlsanity.com/paper/1906.03221/) (2019)
- [Logic2Text: High-Fidelity Natural Language Generation from Logical Forms](https://mlsanity.com/paper/2004.14579/) (2020)
- [Robust (Controlled) Table-to-Text Generation with Structure-Aware Equivariance Learning](https://mlsanity.com/paper/2205.03972/) (2022)
- [Multilingual Pixel Representations for Translation and Effective Cross-lingual Transfer](https://mlsanity.com/paper/2305.14280/) (2023)
- [Neural Pipeline for Zero-Shot Data-to-Text Generation](https://mlsanity.com/paper/2203.16279/) (2022)

## Built on by (most cited first, 6 of 6)

- [Large Language Model for Table Processing: A Survey](https://mlsanity.com/paper/2402.05121/) (2024)
- [Improving Language Understanding from Screenshots](https://mlsanity.com/paper/2402.14073/) (2024)
- [Texts or Images? A Fine-grained Analysis on the Effectiveness of Input Representations and Models for Table Question Answering](https://mlsanity.com/paper/2505.14131/) (2025)
- [TABLET: A Large-Scale Dataset for Robust Visual Table Understanding](https://mlsanity.com/paper/2509.21205/) (2025)
- [Integrating Large Language Models with Graph-based Reasoning for Conversational Question Answering](https://mlsanity.com/paper/2407.09506/) (2024)
- [A Table Is Worth 64 Tokens: Pixel-level Compression for Multi-Table Document Question Answering](https://mlsanity.com/paper/2608.26949/) (2026)

## Abstract (arXiv)

Table-to-text generation involves generating appropriate textual descriptions given structured tabular data. It has attracted increasing attention in recent years thanks to the popularity of neural network models and the availability of large-scale datasets. A common feature across existing methods is their treatment of the input as a string, i.e., by employing linearization techniques that do not always preserve information in the table, are verbose, and lack space efficiency. We propose to rethink data-to-text generation as a visual recognition task, removing the need for rendering the input in a string format. We present PixT3, a multimodal table-to-text model that overcomes the challenges of linearization and input size limitations encountered by existing models. PixT3 is trained with a new self-supervised learning objective to reinforce table structure awareness and is applicable to open-ended and controlled generation settings. Experiments on the ToTTo and Logic2Text benchmarks show that PixT3 is competitive and, in some settings, superior to generators that operate solely on text.