mlsanityData to 7 Oct 2026

COPR: Continual Learning Human Preference through Optimal Policy Regularization

arXiv 2310.15694 ↗First posted 24 Oct 2023cs.LG cs.CLcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The technique of Reinforcement Learning from Human Feedback (RLHF) is a commonly employed method to improve pre-trained Language Models (LM), enhancing their ability to conform to human preferences. Nevertheless, the current RLHF-based LMs necessitate full retraining each time novel queries or feedback are introduced, which becomes a challenging task because human preferences can vary between different domains or tasks. Retraining LMs poses practical difficulties in many real-world situations due to the significant time and computational resources required, along with concerns related to data privacy. To address this limitation, we propose a new method called Continual Optimal Policy Regularization (COPR), in which we compute the distribution of optimal policy bypassing the partition function and then regularize the current policy based on the historically optimal distribution to mitigate Catastrophic Forgetting (CF). COPR involves a single learning phase and doesn't necessitate complex reinforcement learning. Importantly, it shares the capability with RLHF to learn from unlabeled data by maintaining a scoring module, similar to reward model, making it flexible for continually learning without human feedback. Our experimental results show that COPR outperforms strong Continuous Learning (CL) baselines when it comes to consistently aligning with human preferences on incremental tasks and domains.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 40 of its 54 references that are in the map. First posted on arXiv 24 Oct 2023.

Builds on

40 of 54 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignment30 Sep
  2. 2023Stabilizing RLHF through Advantage Model and Selective Rehearsal18 Sep
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  5. 2022Models of human preference for learning reward functions5 Jun
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2020Learning to summarize from human feedbackcited by 2,562
  5. 2016Learning without Forgettingcited by 2,465
  6. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  7. 2016Progressive Neural Networkscited by 1,332
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  9. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  10. 2022Red Teaming Language Models with Language Modelscited by 983
  11. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
21 more
  1. 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 734
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  3. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  4. 2018End-to-End Incremental Learningcited by 596
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  6. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  7. 2018Task-Free Continual Learningcited by 219
  8. 2019Meta-Learning Representations for Continual Learningcited by 189
  9. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  10. 2017Interactive Learning from Policy-Dependent Human Feedbackcited by 155
  11. 2020Adversarial Continual Learningcited by 140
  12. 2021An Empirical Investigation of the Role of Pre-training in Lifelong Learningcited by 134
  13. 2017Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spacescited by 122
  14. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  15. 2020Generative Feature Replay For Class-Incremental Learningcited by 109
  16. 2021Dueling RL: Reinforcement Learning with Trajectory Preferencescited by 105
  17. 2020Safe Imitation Learning via Fast Bayesian Reward Inference from Preferencescited by 73
  18. 2021Self-Supervised Training Enhances Online Continual Learningcited by 53
  19. 2020ECONET: Effective Continual Pretraining of Language Models for Event Temporal Reasoningcited by 38
  20. 2018Memory-based Parameter Adaptationcited by 35
  21. 2013Learning Trajectory Preferences for Manipulators via Iterative Improvementcited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  2. 2026Attribution-Guided Continual Learning for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024COPR: Continual Human Preference Learning via Optimal Policy Regularization
  2. 2025Pref-GUIDE: Continual Policy Learning from Real-Time Human Feedback via Preference-Based Learning
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actions
  5. 2025Policy-labeled Preference Learning: Is Preference Enough for RLHF?
  6. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs
  7. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  8. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs