# An Analysis and Mitigation of the Reversal Curse

arXiv: https://arxiv.org/abs/2311.07468 · HTML: https://mlsanity.com/paper/2311.07468/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 37 references that are in the map.
- First posted on arXiv 13 Nov 2023.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [GLM: General Language Model Pretraining with Autoregressive Blank Infilling](https://mlsanity.com/paper/2103.10360/) (2021)
- [GLM-130B: An Open Bilingual Pre-trained Model](https://mlsanity.com/paper/2210.02414/) (2022)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [Extending Context Window of Large Language Models via Positional Interpolation](https://mlsanity.com/paper/2306.15595/) (2023)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Circuit Component Reuse Across Tasks in Transformer Language Models](https://mlsanity.com/paper/2310.08744/) (2023)
- [Evaluating the Reversal Curse in Model Editing](https://mlsanity.com/paper/2310.10322/) (2023)

## Built on by (most cited first, 22 of 22)

- [LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders](https://mlsanity.com/paper/2404.05961/) (2024)
- [Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey](https://mlsanity.com/paper/2402.05391/) (2024)
- [Reverse Training to Nurse the Reversal Curse](https://mlsanity.com/paper/2403.13799/) (2024)
- [Interpreting Key Mechanisms of Factual Recall in Transformer-Based Language Models](https://mlsanity.com/paper/2403.19521/) (2024)
- [Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use](https://mlsanity.com/paper/2312.04455/) (2023)
- [Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics](https://mlsanity.com/paper/2405.04669/) (2024)
- [The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More](https://mlsanity.com/paper/2406.05183/) (2024)
- [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026)
- [Mixture of In-Context Experts Enhance LLMs' Long Context Awareness](https://mlsanity.com/paper/2406.19598/) (2024)
- [HoPE: A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation](https://mlsanity.com/paper/2410.21216/) (2024)
- [Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure](https://mlsanity.com/paper/2504.01928/) (2025)
- [Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training](https://mlsanity.com/paper/2403.00758/) (2024)
- [Delving into the Reversal Curse: How Far Can Large Language Models Generalize?](https://mlsanity.com/paper/2410.18808/) (2024)
- [Chain of History: Learning and Forecasting with LLMs for Temporal Knowledge Graph Completion](https://mlsanity.com/paper/2401.06072/) (2024)
- [Large Language Models are Limited in Out-of-Context Knowledge Reasoning](https://mlsanity.com/paper/2406.07393/) (2024)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [Enhancing next token prediction based pre-training for jet foundation models](https://mlsanity.com/paper/2512.04149/) (2025)
- [Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs](https://mlsanity.com/paper/2510.09885/) (2025)
- [Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge](https://mlsanity.com/paper/2602.02470/) (2026)
- [Dual-objective Language Models: Training Efficiency Without Overfitting](https://mlsanity.com/paper/2512.14549/) (2025)
- [Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs](https://mlsanity.com/paper/2601.02931/) (2026)
- [The Illusion of Latent Generalization: Bi-directionality and the Reversal Curse](https://mlsanity.com/paper/2604.04943/) (2026)

## Abstract (arXiv)

Recent research observed a noteworthy phenomenon in large language models (LLMs), referred to as the ``reversal curse.'' The reversal curse is that when dealing with two entities, denoted as a and b, connected by their relation R and its inverse R-1, LLMs excel in handling sequences in the form of ``aRb,'' but encounter challenges when processing ``bR-1a,'' whether in generation or comprehension. For instance, GPT-4 can accurately respond to the query ``Tom Cruise's mother is?'' with ``Mary Lee Pfeiffer,'' but it struggles to provide a satisfactory answer when asked ``Mary Lee Pfeiffer's son is?'' In this paper, we undertake the first-ever study of how the reversal curse happens in LLMs. Our investigations reveal that the reversal curse can stem from the specific training objectives, which become particularly evident in the widespread use of next-token prediction within most causal language models. We hope this initial investigation can draw more attention to the reversal curse, as well as other underlying limitations in current LLMs.