mlsanityData to 7 Oct 2026

Exploring the impact of low-rank adaptation on the performance, efficiency, and regularization of RLHF

arXiv 2309.09055 ↗First posted 16 Sep 2023cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

During the last stage of RLHF, a large language model is aligned to human intents via PPO training, a process that generally requires large-scale computational resources. In this technical report, we empirically investigate an efficient implementation of RLHF using low-rank adaptation (LoRA), which allows us to align the LLaMA 7B checkpoint on the Alpaca dataset using only two A100 GPUs instead of the eight required for full model fine-tuning. Despite tuning only 0.2% of LLaMA 7B's parameters, our implementation achieves better performance than the publicly-released AlpacaFarm checkpoint with full model fine-tuning. Next, we analyze several configurations of our LoRA-based PPO implementation, varying the form of the KL regularization term in the training objective. We find that (1) removing this penalty term does not harm performance on the AlpacaFarm evaluation set under our LoRA setup; (2) other regularizers, such as Jensen-Shannon divergence, lead to improved performance; and (3) while PPO training negatively impacts the factuality of model-generated responses, training with LoRA largely mitigates this effect. We release our code and pretrained checkpoints to facilitate future research on more efficient RLHF.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 37 of its 46 references that are in the map. First posted on arXiv 16 Sep 2023.

Builds on

37 of 46 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Efficient RLHF: Reducing the Memory Usage of PPO1 Sep
  2. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  3. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Training2 Jun
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 31

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2015Trust Region Policy Optimizationcited by 2,503
19 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  3. 2023LIMA: Less Is More for Alignmentcited by 1,080
  4. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  7. 2023How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resourcescited by 405
  8. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  9. 2022Self-critiquing models for assisting human evaluatorscited by 372
  10. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  12. 2023Self-Alignment with Instruction Backtranslationcited by 149
  13. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  14. 2023Aligning Language Models with Preferences through f-divergence Minimizationcited by 118
  15. 2020A Distributional Approach to Controlled Text Generationcited by 117
  16. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 80
  17. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  18. 2023Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generationcited by 55
  19. 2023Peering Through Preferences: Unraveling Feedback Acquisition for Aligning Large Language Modelscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 10; 2025: 4; 2026: 2.

20 papers.

Sort:

By area:

  1. 2024LoRA Learns Less and Forgets Lesscited by 313
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 227
  4. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  5. 2024Decoding-time Realignment of Language Modelscited by 67
  6. 2024A Survey on LoRA of Large Language Modelscited by 67
  7. 2023Group Preference Optimization: Few-Shot Alignment of Large Language Modelscited by 61
  8. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  9. 2024Prototypical Reward Network for Data-Efficient RLHFcited by 22
  10. 2025AuroRA: Breaking Low-Rank Bottleneck of LoRA with Nonlinear Mappingcited by 22
  11. 2024Recovering the Pre-Fine-Tuning Weights of Generative Modelscited by 15
  12. 2025Aligned but Blind: Alignment Increases Implicit Bias by Reducing Awareness of Racecited by 15
8 more
  1. 2023Chatmap : Large Language Model Interaction with Cartographic Datacited by 7
  2. 2024Rethinking the Role of Proxy Rewards in Language Model Alignmentcited by 7
  3. 2024LoRTA: Low Rank Tensor Adaptation of Large Language Modelscited by 5
  4. 2024Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Modelscited by 4
  5. 2025A Shared Low-Rank Adaptation Approach to Personalized RLHFcited by 4
  6. 2025What-If Analysis of Large Language Models: Explore the Game World Using Proactive Thinkingcited by 3
  7. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  8. 2026Beyond LoRA vs. Full Fine-Tuning: Gradient-Guided Optimizer Routing for LLM Adaptationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Strategic Over-Parameterization for Generalizable Low-Rank Adaptation
  2. 2025LoRA Is Slower Than You Think
  3. 2024Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
  4. 2025Less is More: Resource-Efficient Low-Rank Adaptation
  5. 2025ScaLoRA: Optimally Scaled Low-Rank Adaptation for Efficient High-Rank Fine-Tuning
  6. 2025RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
  7. 2025GoRA: Gradient-driven Adaptive Low Rank Adaptation
  8. 2025Uni-LoRA: One Vector is All You Need