# OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering

arXiv: https://arxiv.org/abs/2207.03637 · HTML: https://mlsanity.com/paper/2207.03637/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 27 references that are in the map.
- First posted on arXiv 8 Jul 2022.

## Builds on (most cited first, 23 of 23)

- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT](https://mlsanity.com/paper/2004.12832/) (2020)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [TAPEX: Table Pre-training via Learning a Neural SQL Executor](https://mlsanity.com/paper/2107.07653/) (2021)
- [GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing](https://mlsanity.com/paper/2009.13845/) (2020)
- [Structure-Grounded Pretraining for Text-to-SQL](https://mlsanity.com/paper/2010.12773/) (2020)
- [COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List](https://mlsanity.com/paper/2104.07186/) (2021)
- [TUTA: Tree-based Transformers for Generally Structured Table Pre-training](https://mlsanity.com/paper/2010.12537/) (2020)
- [Pre-training via Paraphrasing](https://mlsanity.com/paper/2006.15020/) (2020)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [Learning Contextual Representations for Semantic Parsing with Generation-Augmented Pre-Training](https://mlsanity.com/paper/2012.10309/) (2020)
- [Few-Shot Question Answering by Pretraining Span Selection](https://mlsanity.com/paper/2101.00438/) (2021)
- [UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering](https://mlsanity.com/paper/2012.14610/) (2020)
- [On the Potential of Lexico-logical Alignments for Semantic Parsing to SQL Queries](https://mlsanity.com/paper/2010.11246/) (2020)
- [Memory Augmented Policy Optimization for Program Synthesis and Semantic Parsing](https://mlsanity.com/paper/1807.02322/) (2018)
- [Learning Semantic Parsers from Denotations with Latent Structured Alignments and Abstract Programs](https://mlsanity.com/paper/1909.04165/) (2019)
- [Topic Transferable Table Question Answering](https://mlsanity.com/paper/2109.07377/) (2021)

## Built on by (most cited first, 29 of 29)

- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [LEVER: Learning to Verify Language-to-Code Generation with Execution](https://mlsanity.com/paper/2302.08468/) (2023)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understanding](https://mlsanity.com/paper/2401.04398/) (2024)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [TableGPT2: A Large Multimodal Model with Tabular Data Integration](https://mlsanity.com/paper/2411.02059/) (2024)
- [Enhancing Few-shot Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies](https://mlsanity.com/paper/2305.12586/) (2023)
- [RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations](https://mlsanity.com/paper/2306.14321/) (2023)
- [ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples](https://mlsanity.com/paper/2210.12374/) (2022)
- [A Survey of Table Reasoning with Large Language Models](https://mlsanity.com/paper/2402.08259/) (2024)
- [OpenTab: Advancing Large Language Models as Open-domain Table Reasoners](https://mlsanity.com/paper/2402.14361/) (2024)
- [Effective Distillation of Table-based Reasoning Ability from LLMs](https://mlsanity.com/paper/2309.13182/) (2023)
- [Tree-of-Table: Unleashing the Power of LLMs for Enhanced Large-Scale Table Understanding](https://mlsanity.com/paper/2411.08516/) (2024)
- [S3Eval: A Synthetic, Scalable, Systematic Evaluation Suite for Large Language Models](https://mlsanity.com/paper/2310.15147/) (2023)
- [SheetAgent: Towards A Generalist Agent for Spreadsheet Reasoning and Manipulation via Large Language Models](https://mlsanity.com/paper/2403.03636/) (2024)
- [From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuning](https://mlsanity.com/paper/2304.07995/) (2023)
- [Generate, Transform, Answer: Question Specific Tool Synthesis for Tabular Data](https://mlsanity.com/paper/2303.10138/) (2023)
- [Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks](https://mlsanity.com/paper/2310.00789/) (2023)
- [Understanding and Improving Zero-shot Multi-hop Reasoning in Generative Question Answering](https://mlsanity.com/paper/2210.04234/) (2022)
- [From Rows to Reasoning: A Retrieval-Augmented Multimodal Framework for Spreadsheet Understanding](https://mlsanity.com/paper/2601.08741/) (2026)
- [PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development](https://mlsanity.com/paper/2301.09715/) (2023)
- [Comparative Study of Domain-adapted VLMs for General Document Visual Question Answering](https://mlsanity.com/paper/2607.07179/) (2026)
- [TACR: A Table-alignment-based Cell-selection and Reasoning Model for Hybrid Question-Answering](https://mlsanity.com/paper/2305.14682/) (2023)
- [Training Table Question Answering via SQL Query Decomposition](https://mlsanity.com/paper/2402.13288/) (2024)
- [HCT-QA: A Benchmark for Question Answering on Human-Centric Tables](https://mlsanity.com/paper/2504.20047/) (2025)
- [FACTS: Table Summarization via Offline Template Generation with Agentic Workflows](https://mlsanity.com/paper/2510.13920/) (2025)
- [TabIQA: Table Questions Answering on Business Document Images](https://mlsanity.com/paper/2303.14935/) (2023)
- [Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning](https://mlsanity.com/paper/2305.07475/) (2023)
- [MRT at IberLEF-2025 PRESTA Task: Maximizing Recovery from Tables with Multiple Steps](https://mlsanity.com/paper/2507.12981/) (2025)

## Abstract (arXiv)

The information in tables can be an important complement to text, making table-based question answering (QA) systems of great value. The intrinsic complexity of handling tables often adds an extra burden to both model design and data annotation. In this paper, we aim to develop a simple table-based QA model with minimal annotation effort. Motivated by the fact that table-based QA requires both alignment between questions and tables and the ability to perform complicated reasoning over multiple table elements, we propose an omnivorous pretraining approach that consumes both natural and synthetic data to endow models with these respective abilities. Specifically, given freely available tables, we leverage retrieval to pair them with relevant natural sentences for mask-based pretraining, and synthesize NL questions by converting SQL sampled from tables for pretraining with a QA loss. We perform extensive experiments in both few-shot and full settings, and the results clearly demonstrate the superiority of our model OmniTab, with the best multitasking approach achieving an absolute gain of 16.2% and 2.7% in 128-shot and full settings respectively, also establishing a new state-of-the-art on WikiTableQuestions. Detailed ablations and analyses reveal different characteristics of natural and synthetic data, shedding light on future directions in omnivorous pretraining. Code, pretraining data, and pretrained models are available at https://github.com/jzbjyb/OmniTab.