mlsanityData to 14 Oct 2026

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"

arXiv 2309.12288 ↗First posted 21 Sep 2023cs.CL cs.AI cs.LGcited by 407

Not on the map. Why?

Cited by 81 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We expose a surprising failure of generalization in auto-regressive large language models (LLMs). If a model is trained on a sentence of the form "A is B", it will not automatically generalize to the reverse direction "B is A". This is the Reversal Curse. For instance, if a model is trained on "Valentina Tereshkova was the first woman to travel to space", it will not automatically be able to answer the question, "Who was the first woman to travel to space?". Moreover, the likelihood of the correct answer ("Valentina Tershkova") will not be higher than for a random name. Thus, models do not generalize a prevalent pattern in their training set: if "A is B" occurs, "B is A" is more likely to occur. It is worth noting, however, that if "A is B" appears in-context, models can deduce the reverse relationship. We provide evidence for the Reversal Curse by finetuning GPT-3 and Llama-1 on fictitious statements such as "Uriah Hawthorne is the composer of Abyssal Melodies" and showing that they fail to correctly answer "Who composed Abyssal Melodies?". The Reversal Curse is robust across model sizes and model families and is not alleviated by data augmentation. We also evaluate ChatGPT (GPT-3.5 and GPT-4) on questions about real-world celebrities, such as "Who is Tom Cruise's mother? [A: Mary Lee Pfeiffer]" and the reverse "Who is Mary Lee Pfeiffer's son?". GPT-4 correctly answers questions like the former 79% of the time, compared to 33% for the latter. Code available at: https://github.com/lukasberglund/reversal_curse.

Key facts. Cited by 407 papers in the map (data through 14 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

28 of 41 references.

References · 28

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  7. 2019Language Models as Knowledge Bases?cited by 1,936
  8. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  9. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  10. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  11. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  12. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
16 more
  1. 2023Large Language Modelscited by 780
  2. 2021Editing Factual Knowledge in Language Modelscited by 553
  3. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  4. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  5. 2021Fast Model Editing at Scalecited by 498
  6. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  7. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 449
  8. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  9. 2023Studying Large Language Model Generalization with Influence Functionscited by 326
  10. 2020Modifying Memories in Transformer Modelscited by 224
  11. 2018Contextual Augmentation: Data Augmentation by Words with Paradigmatic Relationscited by 201
  12. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  13. 2021Understanding by Understanding Not: Modeling Negation in Language Modelscited by 73
  14. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  15. 2022Kformer: Knowledge Injection in Transformer Feed-Forward Layerscited by 37
  16. 2020Data Manipulation: Towards Effective Instance Learning for Neural Dialogue Generation via Learning to Augment and Reweightcited by 34

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 27; 2024: 125; 2025: 142; 2026: 113.

407 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  2. 2025Large Language Diffusion Modelscited by 912
  3. 2024Your Absorbing Discrete Diffusion Secretly Models the Conditional Distributions of Clean Datacited by 381
  4. 2024Alignment faking in large language modelscited by 360
  5. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 300
  6. 2025Why Language Models Hallucinatecited by 220
  7. 2024Chain-of-Thought Reasoning Without Promptingcited by 210
  8. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
  9. 2024Scaling up Masked Diffusion Models on Textcited by 199
  10. 2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
  11. 2024Do Large Language Models Latently Perform Multi-Hop Reasoning?cited by 169
  12. 2025dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Cachingcited by 169
38 more
  1. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 163
  2. 2024GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solverscited by 158
  3. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  4. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  5. 2024Multi-Step Reasoning with Large Language Models, a Surveycited by 143
  6. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 136
  7. 2024Make Your LLM Fully Utilize the Contextcited by 119
  8. 2023DyVal: Dynamic Evaluation of Large Language Models for Reasoning Taskscited by 101
  9. 2023The Generative AI Paradox: "What It Can Create, It May Not Understand"cited by 98
  10. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  11. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  12. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Surveycited by 83
  13. 2025Path Planning for Masked Diffusion Model Samplingcited by 80
  14. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79
  15. 2023Thread of Thought Unraveling Chaotic Contextscited by 72
  16. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  17. 2025Are DeepSeek R1 And Other Reasoning Models More Faithful?cited by 72
  18. 2024Alice in Wonderland: Simple Tasks Showing Complete Reasoning Breakdown in State-Of-the-Art Large Language Modelscited by 68
  19. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  20. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
  21. 2024Safetywashing: Do AI Safety Benchmarks Actually Measure Safety Progress?cited by 61
  22. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  23. 2024Synthetic continued pretrainingcited by 60
  24. 2023MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluationcited by 57
  25. 2024Instruction-tuned Language Models are Better Knowledge Learnerscited by 55
  26. 2024Dynamic Evaluation of Large Language Models by Meta Probing Agentscited by 54
  27. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  28. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  29. 2024Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklistcited by 48
  30. 2025Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principlescited by 45
  31. 2024Evolving Code with A Large Language Modelcited by 43
  32. 2024CivRealm: A Learning and Reasoning Odyssey in Civilization for Decision-Making Agentscited by 42
  33. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 42
  34. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  35. 2024Who's asking? User personas and the mechanics of latent misalignmentcited by 40
  36. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
  37. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  38. 2025LongLLaDA: Unlocking Long Context Capabilities in Diffusion LLMscited by 40

50 most cited of 407 All 407 (JSON)