mlsanityData to 7 Oct 2026

Repeated post-training is not Self-improving: Diagnosing Scientific Amnesia in Continual DPO Pipelines

arXiv 2606.21089 ↗First posted 17 Jun 2026cs.AIcited by 1

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

Industrial LLM teams often ship behavior updates by repeatedly DPO-training a base model on sequences of related preference-data campaigns. The dominant failure mode in this regime is not always classical catastrophic forgetting: a pipeline may preserve previously learned behaviors while still failing to accumulate reusable methodological knowledge about how to train the next campaign. We call this failure mode scientific amnesia. This paper turns that practitioner intuition into a measurable industrial problem. We contribute: (i) a diagnostic suite for amnesia, (ii) a Program-based pipeline that chains FSDP-sharded DPO checkpoints across Qwen2.5-7B-Instruct runs, (iii) a 30-campaign HumanEval subdomain benchmark, and (iv) a comparative diagnostic study of five strategy proposers: random memory, rule-based scheduling, retrieval-only memory, warm-start Bayesian optimization, and MSCL, a meta-scientific memory and reasoner candidate. Across a single-seed 5-condition * 3-step real-LM chain, 4 of 5 candidates degrade in step-level peak pass@1, including MSCL; only the deliberately conservative rule-based schedule improves. Follow-up pilots qualify rather than overturn this finding: in a heterogeneous chain, MSCL is the only completed candidate that improves, whereas in a small multi-seed homogeneous sweep, retrieval-only has the best mean Delta and no pairwise candidate gap is statistically distinguishable. The contribution is therefore diagnostic, not a claim that MSCL solves the problem: scientific amnesia is observable in a production-like continual-DPO pipeline, and conclusions about interventions depend sharply on chain regime, evaluator design, and seed coverage.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 20 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

18 of 20 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Understanding Reference Policies in Direct Preference Optimization18 Jul
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  3. 2024ORPO: Monolithic Preference Optimization without Reference Model12 Mar
  4. 2023Mitigating the Alignment Tax of RLHF12 Sep
  5. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 13

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  5. 2024Qwen2.5 Technical Reportcited by 4,713
  6. 2024A Survey on LLM-as-a-Judgecited by 1,384
  7. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  8. 2018Experience Replay for Continual Learningcited by 1,074
  9. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  10. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  11. 2024Continual Learning for Large Language Models: A Surveycited by 144
  12. 2020AutoML-Zero: Evolving Machine Learning Algorithms From Scratchcited by 137
1 more
  1. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Recursive Self-Improvement in AI: From Bounded Self-Refinement to Autonomous Research Loopscited by 25

Similar, not cited

Close in wording, no citation either way

  1. 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
  2. 2026MemGuard: Persisting Verifier Signals for LLM-Agent Memory Governance
  3. 2026The Epistemics of Agent Memory: Measuring, and Governing, the Consolidation Decision in Long-Horizon LLM Agents
  4. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgetting
  5. 2026Pretrained Vision-Language-Action Models are Surprisingly Resistant to Forgetting in Continual Learning
  6. 2026Overcoming Catastrophic Forgetting in Visual Continual Learning with Reinforcement Fine-Tuning
  7. 2026CONTRAMEM: Learning Self-Evolving Procedural Memory from Contrasting Multi-Model Trajectories
  8. 2026Sentry: Learning to Recover from LLM Agent Failures at Test Time