mlsanityData to 14 Oct 2026

Reverse Training to Nurse the Reversal Curse

arXiv 2403.13799 ↗First posted 20 Mar 2024cs.CL cs.AIcited by 49

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have a surprising failure: when trained on "A has a feature B", they do not generalize to "B is a feature of A", which is termed the Reversal Curse. Even when training with trillions of tokens this issue still appears due to Zipf's law - hence even if we train on the entire internet. This work proposes an alternative training scheme, called reverse training, whereby all words are used twice, doubling the amount of available tokens. The LLM is trained in both forward and reverse directions by reversing the training strings while preserving (i.e., not reversing) chosen substrings, such as entities. We show that data-matched reverse-trained models provide superior performance to standard models on standard tasks, and compute-matched reverse-trained models provide far superior performance on reversal tasks, helping resolve the reversal curse issue.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

25 of 33 references.

References · 25

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  9. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  10. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  11. 2019Cross-lingual Language Model Pretrainingcited by 1,090
  12. 2023The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"cited by 407
13 more
  1. 2023Physics of Language Models: Part 3.1, Knowledge Storage and Extractioncited by 270
  2. 2022Efficient Training of Language Models to Fill in the Middlecited by 255
  3. 2023Language Modeling Is Compressioncited by 225
  4. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 177
  5. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 161
  6. 2023Self-Alignment with Instruction Backtranslationcited by 151
  7. 2023Taken out of context: On measuring situational awareness in LLMscited by 143
  8. 2023Physics of Language Models: Part 3.2, Knowledge Manipulationcited by 143
  9. 2020Pre-training via Paraphrasingcited by 92
  10. 2024Repetition Improves Language Model Embeddingscited by 72
  11. 2020FLERT: Document-Level Features for Named Entity Recognitioncited by 45
  12. 2023Meet in the Middle: A New Pre-training Paradigmcited by 22
  13. 2023An Analysis and Mitigation of the Reversal Cursecited by 22

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 22; 2026: 8.

49 papers.

Sort:

By area:

  1. 2025Train for the Worst, Plan for the Best: Understanding Token Ordering in Masked Diffusionscited by 214
  2. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 144
  3. 2024Beyond Autoregression: Discrete Diffusion for Complex Reasoning and Planningcited by 135
  4. 2024Changing Answer Order Can Decrease MMLU Accuracycited by 42
  5. 2025On the generalization of language models from in-context learning and finetuning: a controlled studycited by 40
  6. 2024Are large language models superhuman chemists?cited by 34
  7. 2024Scaling Laws for Fact Memorization of Large Language Modelscited by 33
  8. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  9. 2024The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and Morecited by 25
  10. 2024Causality for Large Language Modelscited by 25
  11. 2024A Survey on the Honesty of Large Language Modelscited by 23
  12. 2026Large Language Model Reasoning Failurescited by 21
37 more
  1. 2024σ-GPTs: A New Approach to Autoregressive Modelscited by 20
  2. 2025Beyond Multi-Token Prediction: Pretraining LLMs with Future Summariescited by 15
  3. 2025Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failurecited by 13
  4. 2024Golden-Retriever: High-Fidelity Agentic Retrieval Augmented Generation for Industrial Knowledge Basecited by 11
  5. 2024Constraint Back-translation Improves Complex Instruction Following of Large Language Modelscited by 11
  6. 2024The Mystery of the Pathological Path-star Task for Language Modelscited by 8
  7. 2025How new data permeates LLM knowledge and how to dilute itcited by 8
  8. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  9. 2025Rote Learning Considered Useful: Generalizing over Memorized Data in LLMscited by 5
  10. 2025Decomposing Complex Visual Comprehension into Atomic Visual Skills for Vision Language Modelscited by 4
  11. 2024Are Large Language Models Strategic Decision Makers? A Study of Performance and Bias in Two-Player Non-Zero-Sum Gamescited by 3
  12. 2024Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Modelscited by 3
  13. 2024Time-Reversal Provides Unsupervised Feedback to LLMscited by 3
  14. 2025Your Language Model May Think Too Rigidly: Achieving Reasoning Consistency with Symmetry-Enhanced Trainingcited by 3
  15. 2025Language Models, Graph Searching, and Supervision Adulteration: When More Supervision is Less and How to Make More Morecited by 3
  16. 2026TABES: Trajectory-Aware Backward-on-Entropy Steering for Masked Diffusion Modelscited by 3
  17. 2024Enhancing elusive clues in knowledge learning by contrasting attention of language modelscited by 2
  18. 2024Reverse Modeling in Large Language Modelscited by 2
  19. 2024Next-Token Prediction Task Assumes Optimal Data Ordering for LLM Training in Proof Generationcited by 2
  20. 2025From Poisoned to Aware: Fostering Backdoor Self-Awareness in LLMscited by 2
  21. 2025Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMscited by 2
  22. 2026Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridgecited by 2
  23. 2025RECALL: Library-Like Behavior In Language Models is Enhanced by Self-Referencing Causal Cyclescited by 1
  24. 2025What Makes the Preferred Thinking Direction for LLMs in Multiple-choice Questions?cited by 1
  25. 2025Open Problems and a Hypothetical Path Forward in LLM Knowledge Paradigmscited by 1
  26. 2025Understanding Fact Recall in Language Models: Why Two-Stage Training Encourages Memorization but Mixed Training Teaches Knowledgecited by 1
  27. 2025LEDOM: Reverse Language Modelcited by 1
  28. 2025Improving LLMs' Learning for Coreference Resolutioncited by 1
  29. 2025Bilinear representation mitigates reversal curse and enables consistent model editingcited by 1
  30. 2025Order Doesn't Matter, But Reasoning Does: Training LLMs with Order-Centric Augmentationcited by 0
  31. 2025VLMs Can Aggregate Scattered Training Patchescited by 0
  32. 2025Reversal Invariance in Autoregressive Language Modelscited by 0
  33. 2026A Theoretical Analysis of Why Masked Diffusion Models Mitigate the Reversal Cursecited by 0
  34. 2026Improving Latent Generalization Using Test-time Computecited by 0
  35. 2026The Illusion of Latent Generalization: Bi-directionality and the Reversal Cursecited by 0
  36. 2026Distilling Directional Verificationcited by 0
  37. 2026Learning a Fact Is Not Learning How to Retrieve Itcited by 0