mlsanityData to 7 Oct 2026

Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning

arXiv 2108.06743 ↗First posted 15 Aug 2021cs.CL cs.AIcited by 17

Not on the map. Why?

Abstract (arXiv)

To quantitatively and intuitively explore the generalization ability of pre-trained language models (PLMs), we have designed several tasks of arithmetic and logical reasoning. We both analyse how well PLMs generalize when the test data is in the same distribution as the train data and when it is different, for the latter analysis, we have also designed a cross-distribution test set other than the in-distribution test set. We conduct experiments on one of the most advanced and publicly released generative PLM - BART. Our research finds that the PLMs can easily generalize when the distribution is the same, however, it is still difficult for them to generalize out of the distribution.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 26 of its 36 references that are in the map. First posted on arXiv 15 Aug 2021.

Builds on

26 of 36 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  8. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  9. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  10. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  11. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  12. 2019COMET: Commonsense Transformers for Automatic Knowledge Graph Constructioncited by 450
14 more
  1. 2020Transformers as Soft Reasoners over Languagecited by 265
  2. 2019Do NLP Models Know Numbers? Probing Numeracy in Embeddingscited by 157
  3. 2020Question and Answer Test-Train Overlap in Open-Domain Question Answering Datasetscited by 137
  4. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  5. 2020Injecting Numerical Reasoning Skills into Language Modelscited by 133
  6. 2020How Can We Accelerate Progress Towards Human-like Linguistic Generalization?cited by 114
  7. 2019BERTs of a feather do not generalize together: Large variability in generalization across models with similar test set performancecited by 89
  8. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  9. 2020Ape210K: A Large-Scale and Template-Rich Dataset of Math Word Problemscited by 60
  10. 2019Does It Make Sense? And Why? A Pilot Study for Sense Making and Explanationcited by 43
  11. 2020SemEval-2020 Task 4: Commonsense Validation and Explanationcited by 42
  12. 2019Measuring Arithmetic Extrapolation Performancecited by 4
  13. 2020Commonsense Knowledge Graph Reasoning by Selection or Generation? Why?cited by 4
  14. 2018Attending to Mathematical Language with Transformerscited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 2; 2023: 5; 2024: 8; 2025: 1.

17 papers.

Sort:

By area:

  1. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  2. 2023How well do Large Language Models perform in Arithmetic tasks?cited by 130
  3. 2023Teaching Arithmetic to Small Transformerscited by 121
  4. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  5. 2023GPT Can Solve Mathematical Problems Without a Calculatorcited by 68
  6. 2024Can Language Models Learn to Skip Steps?cited by 59
  7. 2023Mathematical Language Models: A Surveycited by 24
  8. 2024Mathify: Evaluating Large Language Models on Mathematical Problem Solving Taskscited by 12
  9. 2025A Survey on Mathematical Reasoning and Optimization with Large Language Modelscited by 12
  10. 2022Induced Natural Language Rationales and Interleaved Markup Tokens Enable Extrapolation in Large Language Modelscited by 10
  11. 2024Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Taskscited by 10
  12. 2024Scaling Behavior for Large Language Models regarding Numeral Systems: An Example using Pythiacited by 8
5 more
  1. 2024Quantifying artificial intelligence through algorithmic generalizationcited by 5
  2. 2022ChemAlgebra: Algebraic Reasoning on Chemical Reactionscited by 1
  3. 2024LogicPrpBank: A Corpus for Logical Implication and Equivalencecited by 0
  4. 2024NUMCoT: Numerals and Units of Measurement in Chain-of-Thought Reasoning using Large Language Modelscited by 0
  5. 2024The Extrapolation Power of Implicit Modelscited by 0