mlsanityData to 13 Oct 2026

An Analysis and Mitigation of the Reversal Curse

arXiv 2311.07468 ↗First posted 13 Nov 2023cs.CL cs.AI cs.LGcited by 22

Not on the map. Why?

Abstract (arXiv)

Recent research observed a noteworthy phenomenon in large language models (LLMs), referred to as the ``reversal curse.'' The reversal curse is that when dealing with two entities, denoted as a and b, connected by their relation R and its inverse R-1, LLMs excel in handling sequences in the form of ``aRb,'' but encounter challenges when processing ``bR-1a,'' whether in generation or comprehension. For instance, GPT-4 can accurately respond to the query ``Tom Cruise's mother is?'' with ``Mary Lee Pfeiffer,'' but it struggles to provide a satisfactory answer when asked ``Mary Lee Pfeiffer's son is?'' In this paper, we undertake the first-ever study of how the reversal curse happens in LLMs. Our investigations reveal that the reversal curse can stem from the specific training objectives, which become particularly evident in the widespread use of next-token prediction within most causal language models. We hope this initial investigation can draw more attention to the reversal curse, as well as other underlying limitations in current LLMs.

Key facts. Cited by 22 papers in the map (data through 13 Oct 2026). Builds on 17 of its 37 references that are in the map. First posted on arXiv 13 Nov 2023.

Builds on

17 of 37 references.

References · 17

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,916
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,650
  8. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,145
  9. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,029
  10. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 841
  11. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 833
  12. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 727
5 more
  1. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
  2. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 404
  3. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  4. 2023Circuit Component Reuse Across Tasks in Transformer Language Modelscited by 105
  5. 2023Evaluating the Reversal Curse in Model Editingcited by 34

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 1; 2024: 12; 2025: 5; 2026: 4.

22 papers.

Sort:

By area:

  1. 2024LLM2Vec: Large Language Models Are Secretly Powerful Text Encoderscited by 398
  2. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 51
  3. 2024Reverse Training to Nurse the Reversal Cursecited by 49
  4. 2024Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Modelscited by 30
  5. 2023Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Usecited by 27
  6. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  7. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  8. 2026Large Language Model Reasoning Failurescited by 21
  9. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  10. 2024HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolationcited by 17
  11. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  12. 2024Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Trainingcited by 11
10 more
  1. 2024Delving into the Reversal Curse: How Far Can Large Language Models Generalize?cited by 11
  2. 2024Chain of History: Learning and Forecasting with LLMs for Temporal Knowledge Graph Completioncited by 10
  3. 2024Large Language Models are Limited in Out-of-Context Knowledge Reasoningcited by 6
  4. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  5. 2025Enhancing next token prediction based pre-training for jet foundation modelscited by 3
  6. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2
  7. 2026Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridgecited by 2
  8. 2025Dual-objective Language Models: Training Efficiency Without Overfittingcited by 0
  9. 2026Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMscited by 0
  10. 2026The Illusion of Latent Generalization: Bi-directionality and the Reversal Cursecited by 0