mlsanityData to 7 Oct 2026

Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills

arXiv 2107.07261 ↗First posted 15 Jul 2021cs.CL cs.LGcited by 39

Not on the map. Why?

Abstract (arXiv)

Models pre-trained with a language modeling objective possess ample world knowledge and language skills, but are known to struggle in tasks that require reasoning. In this work, we propose to leverage semi-structured tables, and automatically generate at scale question-paragraph pairs, where answering the question requires reasoning over multiple facts in the paragraph. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills such as number comparison, conjunction, and fact composition. To improve data efficiency, we propose sampling strategies that focus training on reasoning skills the model is currently lacking. We evaluate our approach on three reading comprehension datasets that are focused on reasoning, and show that our model, PReasM, substantially outperforms T5, a popular pre-trained encoder-decoder model. Moreover, sampling examples based on current model errors leads to faster training and higher overall performance.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 57 of its 67 references that are in the map. First posted on arXiv 15 Jul 2021.

Builds on

57 of 67 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  9. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  10. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  11. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  12. 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
38 more
  1. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  2. 2021A Survey of Data Augmentation Approaches for NLPcited by 408
  3. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  4. 2020Transformers as Soft Reasoners over Languagecited by 265
  5. 2017Automated Curriculum Learning for Neural Networkscited by 251
  6. 2021MultiModalQA: Complex Question Answering over Text, Tables and Imagescited by 177
  7. 2021FeTaQA: Free-form Table Question Answeringcited by 174
  8. 2020GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsingcited by 171
  9. 2019oLMpics -- On what Language Model Pre-training Capturescited by 161
  10. 2019Good-Enough Compositional Data Augmentationcited by 158
  11. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  12. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  13. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  14. 2019Distributionally Robust Language Modelingcited by 142
  15. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  16. 2020Training Question Answering Models From Synthetic Datacited by 95
  17. 2020INFOTABS: Inference on Tables as Semi-structured Datacited by 84
  18. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  19. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  20. 2020Logic-Guided Data Augmentation and Regularization for Consistent Question Answeringcited by 77
  21. 2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 77
  22. 2020Balancing Training for Multilingual Neural Machine Translationcited by 71
  23. 2020Understanding tables with intermediate pre-trainingcited by 68
  24. 2019Neural Module Networks for Reasoning over Textcited by 65
  25. 2020Zero-Shot Transfer Learning with Synthesized Data for Multi-Domain Dialogue State Trackingcited by 65
  26. 2020Text Modular Networks: Learning to Decompose Tasks in the Language of Existing Modelscited by 65
  27. 2021Few-Shot Question Answering by Pretraining Span Selectioncited by 62
  28. 2019MultiQA: An Empirical Investigation of Generalization and Transfer in Reading Comprehensioncited by 60
  29. 2019NumNet: Machine Reading Comprehension with Numerical Reasoningcited by 59
  30. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  31. 2020IIRC: A Dataset of Incomplete Information Reading Comprehension Questionscited by 52
  32. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50
  33. 2020Question Directed Graph Attention Network for Numerical Reasoning over Textcited by 35
  34. 2020DeSePtion: Dual Sequence Prediction and Adversarial Examples for Improved Fact-Checkingcited by 26
  35. 2021Incorporating External Knowledge to Enhance Tabular Reasoningcited by 23
  36. 2019Diversify Your Datasets: Analyzing Generalization via Controlled Variance in Adversarial Datasetscited by 21
  37. 2021Database Reasoning Over Textcited by 20
  38. 2021TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-trainingcited by 14

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 18; 2023: 11; 2024: 2; 2025: 3; 2026: 1.

39 papers.

Sort:

By area:

  1. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  2. 2022Large Language Models are Better Reasoners with Self-Verificationcited by 304
  3. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  4. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 231
  5. 2022Binding Language Models in Symbolic Languagescited by 224
  6. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 175
  7. 2022Successive Prompting for Decomposing Complex Questionscited by 94
  8. 2022Reasoning Like Program Executorscited by 45
  9. 2022Table Pre-training: A Survey on Model Architectures, Pre-training Objectives, and Downstream Taskscited by 42
  10. 2023HYTREL: Hypergraph-enhanced Tabular Data Representation Learningcited by 35
  11. 2022On Grounded Planning for Embodied Tasks with Language Modelscited by 33
  12. 2022ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examplescited by 29
27 more
  1. 2022Enhancing Multi-modal and Multi-hop Question Answering via Structured Knowledge and Unified Retrieval-Generationcited by 28
  2. 2023TempTabQA: Temporal Question Answering for Semi-Structured Tablescited by 27
  3. 2023GraphReason: Enhancing Reasoning Capabilities of Large Language Models through A Graph-Based Verification Approachcited by 21
  4. 2021FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretrainingcited by 19
  5. 2023MMHQA-ICL: Multimodal In-context Learning for Hybrid Question Answering over Text, Tables and Imagescited by 18
  6. 2021Teaching Autoregressive Language Models Complex Tasks By Demonstrationcited by 15
  7. 2022LogiGAN: Learning Logical Reasoning via Adversarial Pre-trainingcited by 11
  8. 2023Testing the Limits of Unified Sequence to Sequence LLM Pretraining on Diverse Table Data Taskscited by 10
  9. 2021Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoningcited by 9
  10. 2024Exploring Hybrid Question Answering via Program-based Promptingcited by 8
  11. 2024Enhancing Temporal Understanding in LLMs for Semi-structured Tablescited by 8
  12. 2022Teaching Broad Reasoning Skills for Multi-Step QA by Generating Hard Contextscited by 7
  13. 2022Reflection of Thought: Inversely Eliciting Numerical Reasoning in Language Models via Solving Linear Systemscited by 7
  14. 2022A Survey on Table-and-Text HybridQA: Concepts, Methods, Challenges and Future Directionscited by 7
  15. 2022Few-shot Adaptation Works with UnpredicTable Datacited by 6
  16. 2025Video-R4: Reinforcing Text-Rich Video Reasoning with Visual Ruminationcited by 6
  17. 2025The KoLMogorov Test: Compression by Code Generationcited by 5
  18. 2022Realistic Data Augmentation Framework for Enhancing Tabular Reasoningcited by 4
  19. 2023Unifying Structure Reasoning and Language Model Pre-training for Complex Reasoningcited by 4
  20. 2025Rethinking Information Synthesis in Multimodal Question Answering A Multi-Agent Perspectivecited by 4
  21. 2022Enhancing Tabular Reasoning with Pattern Exploiting Trainingcited by 3
  22. 2022Leveraging Data Recasting to Enhance Tabular Reasoningcited by 3
  23. 2023ReTAG: Reasoning Aware Table to Analytic Text Generationcited by 3
  24. 2023InfoSync: Information Synchronization across Multilingual Semi-structured Tablescited by 3
  25. 2023Teaching Smaller Language Models To Generalise To Unseen Compositional Questionscited by 2
  26. 2023HopPG: Self-Iterative Program Generation for Multi-Hop Question Answering over Heterogeneous Knowledgecited by 1
  27. 2026Evolution of Multimodal Question Answering: From Modality-Adaptive Extraction to Unified Language Representationcited by 0