# The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More

arXiv: https://arxiv.org/abs/2406.05183 · HTML: https://mlsanity.com/paper/2406.05183/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference](https://mlsanity.com/paper/1902.01007/) (2019)
- [SpanBERT: Improving Pre-training by Representing and Predicting Spans](https://mlsanity.com/paper/1907.10529/) (2019)
- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little](https://mlsanity.com/paper/2104.06644/) (2021)
- [CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Text](https://mlsanity.com/paper/1908.06177/) (2019)
- [Physics of Language Models: Part 3.2, Knowledge Manipulation](https://mlsanity.com/paper/2309.14402/) (2023)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [Reverse Training to Nurse the Reversal Curse](https://mlsanity.com/paper/2403.13799/) (2024)
- [Linking artificial and human neural representations of language](https://mlsanity.com/paper/1910.01244/) (2019)
- [ANLIzing the Adversarial Natural Language Inference Dataset](https://mlsanity.com/paper/2010.12729/) (2020)
- [Evaluating Compositionality in Sentence Embeddings](https://mlsanity.com/paper/1802.04302/) (2018)
- [An Analysis and Mitigation of the Reversal Curse](https://mlsanity.com/paper/2311.07468/) (2023)
- [Behavior Analysis of NLI Models: Uncovering the Influence of Three Factors on Robustness](https://mlsanity.com/paper/1805.04212/) (2018)
- [Pre-Training a Language Model Without Human Language](https://mlsanity.com/paper/2012.11995/) (2020)
- [Looking at the Overlooked: An Analysis on the Word-Overlap Bias in Natural Language Inference](https://mlsanity.com/paper/2211.03862/) (2022)

## Built on by (most cited first, 25 of 25)

- [Large Language Diffusion Models](https://mlsanity.com/paper/2502.09992/) (2025)
- [Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusions](https://mlsanity.com/paper/2502.06768/) (2025)
- [Scaling up Masked Diffusion Models on Text](https://mlsanity.com/paper/2410.18514/) (2024)
- [Dualformer: Controllable Fast and Slow Thinking by Learning with Randomized Reasoning Traces](https://mlsanity.com/paper/2410.09918/) (2024)
- [Changing Answer Order Can Decrease MMLU Accuracy](https://mlsanity.com/paper/2406.19470/) (2024)
- [Roll the dice & look before you leap: Going beyond the creative limits of next-token prediction](https://mlsanity.com/paper/2504.15266/) (2025)
- [Language Modeling with Hyperspherical Flows](https://mlsanity.com/paper/2605.11125/) (2026)
- [Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure](https://mlsanity.com/paper/2504.01928/) (2025)
- [Transformers Can Navigate Mazes With Multi-Step Prediction](https://mlsanity.com/paper/2412.05117/) (2024)
- [Looking beyond the next token](https://mlsanity.com/paper/2504.11336/) (2025)
- [Unmasking Trees for Tabular Data](https://mlsanity.com/paper/2407.05593/) (2024)
- [Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More More](https://mlsanity.com/paper/2503.10542/) (2025)
- [Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs](https://mlsanity.com/paper/2510.09885/) (2025)
- [Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridge](https://mlsanity.com/paper/2602.02470/) (2026)
- [On the Trainability of Masked Diffusion Language Models via Blockwise Locality](https://mlsanity.com/paper/2604.24832/) (2026)
- [SiniticMTError: A Machine Translation Dataset with Error Annotations for Sinitic Languages](https://mlsanity.com/paper/2509.20557/) (2025)
- [Bilinear representation mitigates reversal curse and enables consistent model editing](https://mlsanity.com/paper/2509.21993/) (2025)
- [Double Descent as a Lens for Sample Efficiency in Autoregressive vs. Discrete Diffusion Models](https://mlsanity.com/paper/2509.24974/) (2025)
- [Marginals Before Conditionals](https://mlsanity.com/paper/2603.10074/) (2026)
- [Differences in Text Generated by Diffusion and Autoregressive Language Models](https://mlsanity.com/paper/2605.12522/) (2026)
- [Constraints-Guided Diffusion Reasoner for Neuro-Symbolic Learning](https://mlsanity.com/paper/2508.16524/) (2025)
- [A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Curse](https://mlsanity.com/paper/2602.02133/) (2026)
- [BlockGen: Flexible Blockwise Sequence Modeling with Hybrid Samplers](https://mlsanity.com/paper/2606.02241/) (2026)
- [Translation Indeterminacy and the Distributional Fallacy](https://mlsanity.com/paper/2609.07717/) (2026)
- [Distilling Directional Verification](https://mlsanity.com/paper/2610.00997/) (2026)

## Abstract (arXiv)

Today's best language models still struggle with hallucinations: factually incorrect generations, which impede their ability to reliably retrieve information seen during training. The reversal curse, where models cannot recall information when probed in a different order than was encountered during training, exemplifies this in information retrieval. We reframe the reversal curse as a factorization curse - a failure of models to learn the same joint distribution under different factorizations. Through a series of controlled experiments with increasing levels of realism including WikiReversal, a setting we introduce to closely simulate a knowledge intensive finetuning task, we find that the factorization curse is an inherent failure of the next-token prediction objective used in popular large language models. Moreover, we demonstrate reliable information retrieval cannot be solved with scale, reversed tokens, or even naive bidirectional-attention training. Consequently, various approaches to finetuning on specialized data would necessarily provide mixed results on downstream tasks, unless the model has already seen the right sequence of tokens. Across five tasks of varying levels of complexity, our results uncover a promising path forward: factorization-agnostic objectives can significantly mitigate the reversal curse and hint at improved knowledge storage and planning capabilities.