mlsanityData to 7 Oct 2026

Language Model Alignment with Elastic Reset

arXiv 2312.07551 ↗First posted 6 Dec 2023cs.CLcited by 34

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Finetuning language models with reinforcement learning (RL), e.g. from human feedback (HF), is a prominent method for alignment. But optimizing against a reward model can improve on reward while degrading performance in other areas, a phenomenon known as reward hacking, alignment tax, or language drift. First, we argue that commonly-used test metrics are insufficient and instead measure how different algorithms tradeoff between reward and drift. The standard method modified the reward with a Kullback-Lieber (KL) penalty between the online and initial model. We propose Elastic Reset, a new algorithm that achieves higher reward with less drift without explicitly modifying the training objective. We periodically reset the online model to an exponentially moving average (EMA) of itself, then reset the EMA model to the initial model. Through the use of an EMA, our model recovers quickly after resets and achieves higher reward with less drift in the same number of steps. We demonstrate that fine-tuning language models with Elastic Reset leads to state-of-the-art performance on a small scale pivot-translation benchmark, outperforms all baselines in a medium-scale RLHF-like IMDB mock sentiment task and leads to a more performant and more aligned technical QA chatbot with LLaMA-7B. Code available at github.com/mnoukhov/elastic-reset.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 67 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

39 of 67 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar
  2. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
25 more
  1. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2015Trust Region Policy Optimizationcited by 2,503
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  7. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  10. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  11. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  12. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  13. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  14. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  15. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 202
  16. 2020Emergent Multi-Agent Communication in the Deep Learning Eracited by 168
  17. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 100
  18. 2017Findings of the Second Shared Task on Multimodal Machine Translation and Multilingual Image Descriptioncited by 76
  19. 2017Learning with Latent Languagecited by 75
  20. 2019Ease-of-Teaching and Language Structure from Emergent Communicationcited by 65
  21. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 65
  22. 2020Countering Language Drift with Seeded Iterated Learningcited by 57
  23. 2019Countering Language Drift via Visual Groundingcited by 55
  24. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  25. 2022Emergent Communication: Generalization and Overfitting in Lewis Gamescited by 28

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 21; 2025: 8; 2026: 5.

34 papers.

Sort:

By area:

  1. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  2. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  3. 2024A Survey on Self-Evolution of Large Language Modelscited by 86
  4. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  5. 2024SEAL: Safety-enhanced Aligned LLM Fine-tuning via Bilevel Data Selectioncited by 64
  6. 2024Self-Exploring Language Models: Active Preference Elicitation for Online Alignmentcited by 63
  7. 2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
  8. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  9. 2024Coevolving with the Other You: Fine-Tuning LLM with Sequential Cooperative Multi-Agent Reinforcement Learningcited by 32
  10. 2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
  11. 2024Learning to Summarize from LLM-generated Feedbackcited by 24
  12. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
22 more
  1. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learningcited by 20
  2. 2024The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Modelscited by 20
  3. 2024Neuroplastic Expansion in Deep Reinforcement Learningcited by 17
  4. 2024Diffusion Guided Language Modelingcited by 15
  5. 2024Reward Difference Optimization For Sample Reweighting In Offline RLHFcited by 10
  6. 2024Reward-Augmented Data Enhances Direct Preference Alignment of LLMscited by 10
  7. 2025Avoiding mathbfexp(Rmax) scaling in RLHF through Preference-based Explorationcited by 8
  8. 2026Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewardscited by 8
  9. 2024Understanding Likelihood Over-optimisation in Direct Alignment Algorithmscited by 6
  10. 2025Pretrain Value, Not Reward: Decoupled Value Policy Optimizationcited by 6
  11. 2026Safety Alignment as Continual Learning: Mitigating the Alignment Tax via Orthogonal Gradient Projectioncited by 6
  12. 2026EMA Policy Gradient: Taming Reinforcement Learning for LLMs with EMA Anchor and Top-k KLcited by 4
  13. 2026Learning to Control Summaries with Score Rankingcited by 4
  14. 2025Improving Interactive Diagnostic Ability of a Large Language Model Agent Through Clinical Experience Learningcited by 3
  15. 2025Generative Adversarial Post-Training Mitigates Reward Hacking in Live Human-AI Music Interactioncited by 3
  16. 2025General Exploratory Bonus for Optimistic Exploration in RLHFcited by 2
  17. 2024URL: Universal Referential Knowledge Linking via Task-instructed Representation Compressioncited by 1
  18. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1
  19. 2025TuCo: Measuring the Contribution of Fine-Tuning to Individual Responses of LLMscited by 0
  20. 2025Black Box Deployed -- Functional Criteria for Artificial Moral Agents in the LLM Eracited by 0
  21. 2025Heterogeneity in Multi-Agent Reinforcement Learningcited by 0
  22. 2026Learning Where It Matters: Geometric Anchoring for Robust Preference Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Theoretical Limits of Language Model Alignment
  2. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment
  3. 2023Aligning Large Language Models through Synthetic Feedback
  4. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  5. 2025Energy-Based Reward Models for Robust Language Model Alignment
  6. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  7. 2024Language Models Resist Alignment: Evidence From Data Compression
  8. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach