mlsanityData to 7 Oct 2026

oLMpics -- On what Language Model Pre-training Captures

arXiv 1912.13283 ↗First posted 31 Dec 2019cs.CL cs.AI cs.LGcited by 161

Not on the map. Why?

Abstract (arXiv)

Recent success of pre-trained language models (LMs) has spurred widespread interest in the language capabilities that they possess. However, efforts to understand whether LM representations are useful for symbolic reasoning tasks have been limited and scattered. In this work, we propose eight reasoning tasks, which conceptually require operations such as comparison, conjunction, and composition. A fundamental challenge is to understand whether the performance of a LM on a task should be attributed to the pre-trained representations or to the process of fine-tuning on the task data. To address this, we propose an evaluation protocol that includes both zero-shot evaluation (no fine-tuning), as well as comparing the learning curve of a fine-tuned LM to the learning curve of multiple controls, which paints a rich picture of the LM capabilities. Our main findings are that: (a) different LMs exhibit qualitatively different reasoning abilities, e.g., RoBERTa succeeds in reasoning tasks where BERT fails completely; (b) LMs do not reason in an abstract manner and are context-dependent, e.g., while RoBERTa can compare ages, it can do so only when the ages are in the typical range of human ages; (c) On half of our reasoning tasks all models fail completely. Our findings and infrastructure can help future work on designing new datasets, models and objective functions for pre-training.

Key facts. Cited by 161 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 54; 2022: 46; 2023: 34; 2024: 16; 2025: 5; 2026: 6.

161 papers.

Sort:

By area:

  1. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  2. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  3. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  4. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  5. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  6. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 519
  7. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  9. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 361
  10. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  11. 2022A Review on Language Models as Knowledge Basescited by 156
  12. 2022Instruction Induction: From Few Examples to Natural Language Task Descriptionscited by 152
38 more
  1. 2021Investigating the Limitations of Transformers with Simple Arithmetic Taskscited by 136
  2. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  3. 2023The Earth is Flat because...: Investigating LLMs' Belief towards Misinformation via Persuasive Conversationcited by 107
  4. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  5. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  6. 2022A Survey of Knowledge Enhanced Pre-trained Language Modelscited by 76
  7. 2022Iteratively Prompt Pre-trained Language Models for Chain of Thoughtcited by 75
  8. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  9. 2021Probing Across Time: What Does RoBERTa Know and When?cited by 73
  10. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  11. 2023Efficient Benchmarking of Language Modelscited by 71
  12. 2023Prompting for Multimodal Hateful Meme Classificationcited by 68
  13. 2022Recitation-Augmented Language Modelscited by 65
  14. 2021Can Generative Pre-trained Language Models Serve as Knowledge Bases for Closed-book QA?cited by 62
  15. 2022LMPriors: Pre-Trained Language Models as Task-Specific Priorscited by 56
  16. 2023Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modelingcited by 56
  17. 2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53
  18. 2021AR-LSAT: Investigating Analytical Reasoning of Textcited by 49
  19. 2021PROST: Physical Reasoning of Objects through Space and Timecited by 48
  20. 2021Prompting Contrastive Explanations for Commonsense Reasoning Taskscited by 45
  21. 2022Event knowledge in large language models: the gap between the impossible and the unlikelycited by 44
  22. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  23. 2021Let's Play Mono-Poly: BERT Can Reveal Words' Polysemy Level and Partitionability into Sensescited by 39
  24. 2021Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skillscited by 39
  25. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  26. 2021The Dangers of Underclaiming: Reasons for Caution When Reporting How NLP Systems Failcited by 38
  27. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
  28. 2022CommunityLM: Probing Partisan Worldviews from Language Modelscited by 36
  29. 2023Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open Worldcited by 35
  30. 2021Knowledge Enhanced Pretrained Language Models: A Compreshensive Surveycited by 33
  31. 2021Back to Square One: Artifact Detection, Training and Commonsense Disentanglement in the Winograd Schemacited by 32
  32. 2021Relational World Knowledge Representation in Contextual Language Models: A Reviewcited by 31
  33. 2022Can Prompt Probe Pretrained Language Models? Understanding the Invisible Risks from a Causal Viewcited by 31
  34. 2021Core Challenges in Embodied Vision-Language Planningcited by 30
  35. 2022GeoMLAMA: Geo-Diverse Commonsense Probing on Multilingual Pre-Trained Language Modelscited by 30
  36. 2021Exploring the Role of BERT Token Representations to Explain Sentence Probing Resultscited by 29
  37. 2023Dissociating language and thought in large language modelscited by 27
  38. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26

50 most cited of 161 All 161 (JSON)