# Delving into the Reversal Curse: How Far Can Large Language Models Generalize? arXiv: https://arxiv.org/abs/2410.18808 ยท HTML: https://mlsanity.com/paper/2410.18808/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 45 of its 55 references that are in the map. - First posted on arXiv 24 Oct 2024. ## Builds on (most cited first, 30 of 45) - [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023) - [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024) - [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021) - [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023) - [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022) - [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020) - [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022) - [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021) - [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022) - [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022) - [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023) - [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013) - [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020) - [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023) - [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022) - [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021) - [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019) - [Deduplicating Training Data Makes Language Models Better](https://mlsanity.com/paper/2107.06499/) (2021) - [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023) - [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022) - [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023) - [Beyond neural scaling laws: beating power law scaling via data pruning](https://mlsanity.com/paper/2206.14486/) (2022) - [Faith and Fate: Limits of Transformers on Compositionality](https://mlsanity.com/paper/2305.18654/) (2023) - [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023) - [Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task](https://mlsanity.com/paper/2210.13382/) (2022) - [The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"](https://mlsanity.com/paper/2309.12288/) (2023) - [Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedback](https://mlsanity.com/paper/2302.12813/) (2023) - [Studying Large Language Model Generalization with Influence Functions](https://mlsanity.com/paper/2308.03296/) (2023) - [Physics of Language Models: Part 3.1, Knowledge Storage and Extraction](https://mlsanity.com/paper/2309.14316/) (2023) - [Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learning](https://mlsanity.com/paper/2305.14160/) (2023) ## Built on by (most cited first, 11 of 11) - [Large Language Model Reasoning Failures](https://mlsanity.com/paper/2602.06176/) (2026) - [Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure](https://mlsanity.com/paper/2504.01928/) (2025) - [Are Transformers Able to Reason by Connecting Separated Knowledge in Training Data?](https://mlsanity.com/paper/2501.15857/) (2025) - [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025) - [Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs](https://mlsanity.com/paper/2510.09885/) (2025) - [Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge](https://mlsanity.com/paper/2602.02470/) (2026) - [Open Problems and a Hypothetical Path Forward in LLM Knowledge Paradigms](https://mlsanity.com/paper/2504.06823/) (2025) - [Bilinear representation mitigates reversal curse and enables consistent model editing](https://mlsanity.com/paper/2509.21993/) (2025) - [Directional Optimization Asymmetry in Transformers: A Synthetic Stress Test](https://mlsanity.com/paper/2511.19997/) (2025) - [Memorization, Emergence, and Explaining Reversal Failures: A Controlled Study of Relational Semantics in LLMs](https://mlsanity.com/paper/2601.02931/) (2026) - [A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse](https://mlsanity.com/paper/2602.02133/) (2026) ## Abstract (arXiv) While large language models (LLMs) showcase unprecedented capabilities, they also exhibit certain inherent limitations when facing seemingly trivial tasks. A prime example is the recently debated "reversal curse", which surfaces when models, having been trained on the fact "A is B", struggle to generalize this knowledge to infer that "B is A". In this paper, we examine the manifestation of the reversal curse across various tasks and delve into both the generalization abilities and the problem-solving mechanisms of LLMs. This investigation leads to a series of significant insights: (1) LLMs are able to generalize to "B is A" when both A and B are presented in the context as in the case of a multiple-choice question. (2) This generalization ability is highly correlated to the structure of the fact "A is B" in the training documents. For example, this generalization only applies to biographies structured in "[Name] is [Description]" but not to "[Description] is [Name]". (3) We propose and verify the hypothesis that LLMs possess an inherent bias in fact recalling during knowledge application, which explains and underscores the importance of the document structure to successful learning. (4) The negative impact of this bias on the downstream performance of LLMs can hardly be mitigated through training alone. These findings offer a novel perspective on interpreting LLMs' generalization through their intrinsic mechanisms and provide insights for developing more effective learning methods. Our code and data are available at https://github.com/alibaba/thinking_bias.git.