mlsanityData to 14 Oct 2026

Injecting Numerical Reasoning Skills into Language Models

arXiv 2004.04487 ↗First posted 9 Apr 2020cs.CLcited by 133

Not on the map. Why?

Cited by 36 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large pre-trained language models (LMs) are known to encode substantial amounts of linguistic information. However, high-level reasoning skills, such as numerical reasoning, are difficult to learn from a language-modeling objective only. Consequently, existing models for numerical reasoning have used specialized architectures with limited flexibility. In this work, we show that numerical reasoning is amenable to automatic data generation, and thus one can inject this skill into pre-trained LMs, by generating large amounts of data, and training in a multi-task setup. We show that pre-training our model, GenBERT, on this data, dramatically improves performance on DROP (49.3 rightarrow 72.3 F1), reaching performance that matches state-of-the-art models of comparable size, while using a simple and general-purpose encoder-decoder architecture. Moreover, GenBERT generalizes well to math word problem datasets, while maintaining high performance on standard RC tasks. Our approach provides a general recipe for injecting skills into large pre-trained LMs, whenever the skill is amenable to automatic data augmentation.

Key facts. Cited by 133 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 26; 2022: 43; 2023: 29; 2024: 19; 2025: 8; 2026: 8.

133 papers.

Sort:

By area:

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  3. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  4. 2022PAL: Program-aided Language Modelscited by 652
  5. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  6. 2022Making Large Language Models Better Reasoners with Step-Aware Verifiercited by 233
  7. 2023REFINER: Reasoning Feedback on Intermediate Representationscited by 205
  8. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 202
  9. 2023SlideVQA: A Dataset for Document Visual Question Answering on Multiple Imagescited by 179
  10. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  11. 2022mGPT: Few-Shot Learners Go Multilingualcited by 142
  12. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 139
38 more
  1. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derenderingcited by 133
  2. 2022A Survey of Deep Learning for Mathematical Reasoningcited by 124
  3. 2022MultiHiertt: Numerical Reasoning over Multi Hierarchical Tabular and Textual Datacited by 121
  4. 2022Semantic Probabilistic Layers for Neuro-Symbolic Learningcited by 103
  5. 2024A Comprehensive Survey of Scientific Large Language Models and Their Applications in Scientific Discoverycited by 96
  6. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  7. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  8. 2022Probing for the Usage of Grammatical Numbercited by 59
  9. 2021TIMEDIAL: Temporal Commonsense Reasoning in Dialogcited by 55
  10. 2022Perturbation Augmentation for Fairer NLPcited by 52
  11. 2022FiNER: Financial Numeric Entity Recognition for XBRL Taggingcited by 50
  12. 2023SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-trainingcited by 47
  13. 2022Reasoning Like Program Executorscited by 45
  14. 2022PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Financecited by 45
  15. 2024Number Cookbook: Number Understanding of Language Models and How to Improve Itcited by 42
  16. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  17. 2022Commonsense Knowledge Reasoning and Generation with Pre-trained Language Models: A Surveycited by 38
  18. 2021Annotating Columns with Pre-trained Language Modelscited by 37
  19. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  20. 2021MWP-BERT: Numeracy-Augmented Pre-training for Math Word Problem Solvingcited by 32
  21. 2022ReasTAP: Injecting Table Reasoning Skills During Pre-training via Synthetic Reasoning Examplescited by 29
  22. 2023LLM-Prop: Predicting Physical And Electronic Properties Of Crystalline Solids From Their Text Descriptionscited by 29
  23. 2023Mathematical Language Models: A Surveycited by 24
  24. 2021Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoningcited by 23
  25. 2021FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretrainingcited by 19
  26. 2023Can neural networks do arithmetic? A survey on the elementary numerical skills of state-of-the-art deep learning modelscited by 18
  27. 2021A Theoretical Analysis of Fine-tuning with Linear Teacherscited by 17
  28. 2021Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoningcited by 17
  29. 2024Path-Consistency with Prefix Enhancement for Efficient Inference in LLMscited by 17
  30. 2021Decrypting Cryptic Crosswords: Semantically Complex Wordplay Puzzles as a Target for NLPcited by 16
  31. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  32. 2021Teaching Autoregressive Language Models Complex Tasks By Demonstrationcited by 15
  33. 2021NumGPT: Improving Numeracy Ability of Generative Pre-trained Modelscited by 15
  34. 2023Evaluating Transformer Language Models on Arithmetic Operations Using Number Decompositioncited by 15
  35. 2021TAPAS at SemEval-2021 Task 9: Reasoning over tables with intermediate pre-trainingcited by 14
  36. 2022ELASTIC: Numerical Reasoning with Adaptive Symbolic Compilercited by 14
  37. 2023From Zero to Hero: Examining the Power of Symbolic Tasks in Instruction Tuningcited by 14
  38. 2021Investigating Numeracy Learning Ability of a Text-to-Text Transfer Modelcited by 13

50 most cited of 133 All 133 (JSON)