mlsanityData to 7 Oct 2026

Physics of Language Models: Part 3.2, Knowledge Manipulation

arXiv 2309.14402 ↗First posted 25 Sep 2023cs.CL cs.AI cs.LGcited by 142

Not on the map. Why?

Cited by 37 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Language models can store vast factual knowledge, yet their ability to flexibly use this knowledge for downstream tasks (e.g., via instruction finetuning) remains questionable. This paper investigates four fundamental knowledge manipulation tasks: retrieval (e.g., "What is person A's attribute X?"), classification (e.g., "Is A's attribute X even or odd?"), comparison (e.g., "Is A greater than B in attribute X?"), and inverse search (e.g., "Which person's attribute X equals T?"). We show that language models excel in knowledge retrieval but struggle even in the simplest classification or comparison tasks unless Chain of Thoughts (CoTs) are employed during both training and inference. Moreover, their performance in inverse knowledge search is virtually 0%, regardless of the prompts. Our primary contribution is a controlled, synthetic experiment that confirms these weaknesses are inherent to language models: they cannot efficiently manipulate knowledge from pre-training data, even when such knowledge is perfectly stored in the models, despite adequate training and sufficient model size. Our findings also apply to modern pretrained language models such as GPT-4, thus giving rise to many Turing tests to distinguish Humans from contemporary AIs.

Key facts. Cited by 142 papers in the map (data through 7 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 25 Sep 2023.

Builds on

23 of 37 references.

References · 23

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2019Language Models as Knowledge Bases?cited by 1,922
  6. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  7. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  8. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  9. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  10. 2023Active Retrieval Augmented Generationcited by 506
  11. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 399
  12. 2021Internet-Augmented Dialogue Generationcited by 217
11 more
  1. 2020Generation-Augmented Retrieval for Open-domain Question Answeringcited by 186
  2. 2023Head-to-Tail: How Knowledgeable are Large Language Models (LLMs)? A.K.A. Will LLMs Replace Knowledge Graphs?cited by 167
  3. 2021Retrieval Augmented Code Generation and Summarizationcited by 148
  4. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  5. 2020Retrieval-Augmented Generation for Code Summarization via Hybrid GNNcited by 89
  6. 2023Physics of Language Models: Part 1, Learning Hierarchical Language Structurescited by 83
  7. 2023ChatGPT versus Traditional Question Answering for Knowledge Graphs: Current Status and Future Directions Towards Knowledge Graph Chatbotscited by 47
  8. 2023Meet in the Middle: A New Pre-training Paradigmcited by 22
  9. 2022COPEN: Probing Conceptual Knowledge in Pre-trained Language Modelscited by 21
  10. 2016Neural Text Generation from Structured Data with Application to the Biography Domaincited by 20
  11. 2019What Does My QA Model Know? Devising Controlled Probes using Expert Knowledgecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 54; 2025: 65; 2026: 17.

142 papers.

Sort:

By area:

  1. 2025Large Language Diffusion Modelscited by 872
  2. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  3. 2024Alignment faking in large language modelscited by 316
  4. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
  5. 2024Chain-of-Thought Reasoning Without Promptingcited by 208
  6. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 165
  7. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  8. 2024The pitfalls of next-token predictioncited by 156
  9. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  10. 2023Rephrase and Respond: Let Large Language Models Ask Better Questions for Themselvescited by 123
  11. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 111
  12. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
38 more
  1. 2024Physics of Language Models: Part 2.1, Grade-School Math and the Hidden Reasoning Processcited by 107
  2. 2024Looking Inward: Language Models Can Learn About Themselves by Introspectioncited by 105
  3. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  4. 2024How Do Large Language Models Acquire Factual Knowledge During Pretraining?cited by 86
  5. 2024Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalizationcited by 78
  6. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  7. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  8. 2023Alignment for Honestycited by 72
  9. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 69
  10. 2024How Large Language Models Encode Context Knowledge? A Layer-Wise Probing Studycited by 64
  11. 2024Synthetic continued pretrainingcited by 56
  12. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  13. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 54
  14. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  15. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  16. 2024Reverse Training to Nurse the Reversal Cursecited by 48
  17. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  18. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 41
  19. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 39
  20. 2024Are large language models superhuman chemists?cited by 34
  21. 2024Scaling Laws for Fact Memorization of Large Language Modelscited by 33
  22. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  23. 2024Source-Aware Training Enables Knowledge Attribution in Language Modelscited by 29
  24. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28
  25. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  26. 2024A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Languagecited by 26
  27. 2024Do Large Language Models Perform Latent Multi-Hop Reasoning without Exploiting Shortcuts?cited by 26
  28. 2025How Is LLM Reasoning Distracted by Irrelevant Context? An Analysis Using a Controlled Benchmarkcited by 26
  29. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  30. 2025Dynamic Parametric Retrieval Augmented Generation for Test-time Knowledge Enhancementcited by 24
  31. 2025How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Trainingcited by 22
  32. 2023Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Taskscited by 21
  33. 2023Making Harmful Behaviors Unlearnable for Large Language Modelscited by 19
  34. 2025FineMedLM-o1: Enhancing Medical Knowledge Reasoning Ability of LLM from Supervised Fine-Tuning to Test-Time Trainingcited by 19
  35. 2024ALPINE: Unveiling the Planning Capability of Autoregressive Learning in Language Modelscited by 18
  36. 2024Representation Shattering in Transformers: A Synthetic Study with Knowledge Editingcited by 18
  37. 2025Variation in Verification: Understanding Verification Dynamics in Large Language Modelscited by 18
  38. 2025Product of Experts with LLMs: Boosting Performance on ARC Is a Matter of Perspectivecited by 17

50 most cited of 142 All 142 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Disentangling Memory and Reasoning Ability in Large Language Models
  2. 2026Adaptive Retrieval helps Reasoning in LLMs -- but mostly if it's not used
  3. 2025Decoding Knowledge in Large Language Models: A Framework for Categorization and Comprehension
  4. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories
  5. 2022Rethinking with Retrieval: Faithful Large Language Model Inference
  6. 2022Discovering Latent Knowledge in Language Models Without Supervision
  7. 2025Reasoning in Action: MCTS-Driven Knowledge Retrieval for Large Language Models
  8. 2026Micro-Macro Retrieval: Reducing Long-Form Hallucination in Large Language Models