mlsanityData to 7 Oct 2026

Limits for Learning with Language Models

arXiv 2306.12213 ↗First posted 21 Jun 2023cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

With the advent of large language models (LLMs), the trend in NLP has been to train LLMs on vast amounts of data to solve diverse language understanding and generation tasks. The list of LLM successes is long and varied. Nevertheless, several recent papers provide empirical evidence that LLMs fail to capture important aspects of linguistic meaning. Focusing on universal quantification, we provide a theoretical foundation for these empirical findings by proving that LLMs cannot learn certain fundamental semantic properties including semantic entailment and consistency as they are defined in formal semantics. More generally, we show that LLMs are unable to learn concepts beyond the first level of the Borel Hierarchy, which imposes severe limits on the ability of LMs, both large and small, to capture many aspects of linguistic meaning. This means that LLMs will continue to operate without formal guarantees on tasks that require entailments and deep linguistic understanding.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 19 of its 59 references that are in the map. First posted on arXiv 21 Jun 2023.

Builds on

19 of 59 references.

References · 19

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2018CoQA: A Conversational Question Answering Challengecited by 674
  6. 2017Exploring Generalization in Deep Learningcited by 595
  7. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2017Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equationscited by 362
  10. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  11. 2018Stress Test Evaluation for Natural Language Inferencecited by 211
  12. 2017Generalization in Deep Learningcited by 146
7 more
  1. 2018On the Practical Computational Power of Finite Precision RNNs for Language Recognitioncited by 117
  2. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  3. 2020UnNatural Language Inferencecited by 68
  4. 2021Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problemcited by 53
  5. 2022Negation, Coordination, and Quantifiers in Contextualized Language Modelscited by 9
  6. 2013On Learnability, Complexity and Stabilitycited by 4
  7. 2018The meaning of "most" for visual question answering modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 6; 2025: 4; 2026: 2.

17 papers.

Sort:

By area:

  1. 2023Challenges and Applications of Large Language Modelscited by 251
  2. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  3. 2023Large Language Models for Compiler Optimizationcited by 52
  4. 2025Reflect, Retry, Reward: Self-Improving LLMs via Reinforcement Learningcited by 32
  5. 2024Easy Problems That LLMs Get Wrongcited by 29
  6. 2025Token embeddings violate the manifold hypothesiscited by 14
  7. 2023Machine-assisted quantitizing designs: augmenting humanities and social sciences with artificial intelligencecited by 13
  8. 2024Enhancing Post-Hoc Attributions in Long Document Comprehension via Coarse Grained Answer Decompositioncited by 12
  9. 2024Exploring Failure Cases in Multimodal Reasoning About Physical Dynamicscited by 5
  10. 2023It Ain't That Bad: Understanding the Mysterious Performance Drop in OOD Generalization for Generative Transformer Modelscited by 2
  11. 2024Fine-tuning and Prompt Engineering with Cognitive Knowledge Graphs for Scholarly Knowledge Organizationcited by 2
  12. 2025Rosetta-PL: Propositional Logic as a Benchmark for Large Language Model Reasoningcited by 2
5 more
  1. 2024Should AI Optimize Your Code? A Comparative Study of Classical Optimizing Compilers Versus Current Large Language Modelscited by 1
  2. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  3. 2026Multilingual Large Language Models do not comprehend all natural languages to equal degreescited by 1
  4. 2023Trustworthy Formal Natural Language Specificationscited by 0
  5. 2025LLM-Augmented Symbolic NLU System for More Reliable Continuous Causal Statement Interpretationcited by 0