# Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

arXiv: https://arxiv.org/abs/2107.07261 · HTML: https://mlsanity.com/paper/2107.07261/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 67 references that are in the map.
- First posted on arXiv 15 Jul 2021.

## Builds on (most cited first, 30 of 57)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs](https://mlsanity.com/paper/1903.00161/) (2019)
- [UnifiedQA: Crossing Format Boundaries With a Single QA System](https://mlsanity.com/paper/2005.00700/) (2020)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [The Web as a Knowledge-base for Answering Complex Questions](https://mlsanity.com/paper/1803.06643/) (2018)
- [A Survey of Data Augmentation Approaches for NLP](https://mlsanity.com/paper/2105.03075/) (2021)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [Automated Curriculum Learning for Neural Networks](https://mlsanity.com/paper/1704.03003/) (2017)
- [MultiModalQA: Complex Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2104.06039/) (2021)
- [FeTaQA: Free-form Table Question Answering](https://mlsanity.com/paper/2104.00369/) (2021)
- [GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing](https://mlsanity.com/paper/2009.13845/) (2020)
- [oLMpics -- On what Language Model Pre-training Captures](https://mlsanity.com/paper/1912.13283/) (2019)
- [Do NLP Models Know Numbers? Probing Numeracy in Embeddings](https://mlsanity.com/paper/1909.07940/) (2019)
- [Good-Enough Compositional Data Augmentation](https://mlsanity.com/paper/1904.09545/) (2019)
- [Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering](https://mlsanity.com/paper/1911.10470/) (2019)
- [Synthetic QA Corpora Generation with Roundtrip Consistency](https://mlsanity.com/paper/1906.05416/) (2019)
- [Distributionally Robust Language Modeling](https://mlsanity.com/paper/1909.02060/) (2019)
- [Injecting Numerical Reasoning Skills into Language Models](https://mlsanity.com/paper/2004.04487/) (2020)
- [Training Question Answering Models From Synthetic Data](https://mlsanity.com/paper/2002.09599/) (2020)
- [INFOTABS: Inference on Tables as Semi-structured Data](https://mlsanity.com/paper/2005.06117/) (2020)
- [Representing Numbers in NLP: a Survey and a Vision](https://mlsanity.com/paper/2103.13136/) (2021)

## Built on by (most cited first, 30 of 39)

- [MuSiQue: Multihop Questions via Single-hop Question Composition](https://mlsanity.com/paper/2108.00573/) (2021)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Making Large Language Models Better Reasoners with Step-Aware Verifier](https://mlsanity.com/paper/2206.02336/) (2022)
- [UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models](https://mlsanity.com/paper/2201.05966/) (2022)
- [Binding Language Models in Symbolic Languages](https://mlsanity.com/paper/2210.02875/) (2022)
- [SlideVQA: A Dataset for Document Visual Question Answering on Multiple Images](https://mlsanity.com/paper/2301.04883/) (2023)
- [Successive Prompting for Decomposing Complex Questions](https://mlsanity.com/paper/2212.04092/) (2022)
- [Reasoning Like Program Executors](https://mlsanity.com/paper/2201.11473/) (2022)
- [Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Tasks](https://mlsanity.com/paper/2201.09745/) (2022)
- [HYTREL: Hypergraph-enhanced Tabular Data Representation Learning](https://mlsanity.com/paper/2307.08623/) (2023)
- [On Grounded Planning for Embodied Tasks with Language Models](https://mlsanity.com/paper/2209.00465/) (2022)
- [ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examples](https://mlsanity.com/paper/2210.12374/) (2022)
- [Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generation](https://mlsanity.com/paper/2212.08632/) (2022)
- [TempTabQA: Temporal Question Answering for Semi-Structured Tables](https://mlsanity.com/paper/2311.08002/) (2023)
- [GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approach](https://mlsanity.com/paper/2308.09267/) (2023)
- [FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining](https://mlsanity.com/paper/2109.07323/) (2021)
- [MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Images](https://mlsanity.com/paper/2309.04790/) (2023)
- [Teaching Autoregressive Language Models Complex Tasks By Demonstration](https://mlsanity.com/paper/2109.02102/) (2021)
- [LogiGAN: Learning Logical Reasoning via Adversarial Pre-training](https://mlsanity.com/paper/2205.08794/) (2022)
- [Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Tasks](https://mlsanity.com/paper/2310.00789/) (2023)
- [Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning](https://mlsanity.com/paper/2108.00578/) (2021)
- [Exploring Hybrid Question Answering via Program-based Prompting](https://mlsanity.com/paper/2402.10812/) (2024)
- [Enhancing Temporal Understanding in LLMs for Semi-structured Tables](https://mlsanity.com/paper/2407.16030/) (2024)
- [Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contexts](https://mlsanity.com/paper/2205.12496/) (2022)
- [Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systems](https://mlsanity.com/paper/2210.05075/) (2022)
- [A Survey on Table-and-Text HybridQA: Concepts, Methods, Challenges and Future Directions](https://mlsanity.com/paper/2212.13465/) (2022)
- [Few-shot Adaptation Works with UnpredicTable Data](https://mlsanity.com/paper/2208.01009/) (2022)
- [Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Rumination](https://mlsanity.com/paper/2511.17490/) (2025)
- [The KoLMogorov Test: Compression by Code Generation](https://mlsanity.com/paper/2503.13992/) (2025)
- [Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspective](https://mlsanity.com/paper/2505.20816/) (2025)

## Abstract (arXiv)

Models pre-trained with a language modeling objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning. In this work, we propose to leverage semi-structured tables, and automatically generate at scale question-paragraph pairs, where answering the question requires reasoning over multiple facts in the paragraph. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills such as number comparison, conjunction, and fact composition. To improve data efficiency, we propose sampling strategies that focus training on reasoning skills the model is currently lacking. We evaluate our approach on three reading comprehension datasets that are focused on reasoning, and show that our model, PReasM, substantially outperforms T5, a popular pre-trained encoder-decoder model. Moreover, sampling examples based on current model errors leads to faster training and higher overall performance.