mlsanityData to 7 Oct 2026

Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy

arXiv 2403.04283 ↗First posted 7 Mar 2024cs.CL cs.AI cs.LGcited by 2

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Reinforcement Learning from Human Feedback (RLHF) is the prevailing approach to ensure Large Language Models (LLMs) align with human values. However, existing RLHF methods require a high computational cost, one main reason being that RLHF assigns both the generation and alignment tasks to the LLM simultaneously. In this paper, we introduce Proxy-RLHF, which decouples the generation and alignment processes of LLMs, achieving alignment with human values at a much lower computational cost. We start with a novel Markov Decision Process (MDP) designed for the alignment process and employ Reinforcement Learning (RL) to train a streamlined proxy model that oversees the token generation of the LLM, without altering the LLM itself. Experiments show that our method achieves a comparable level of alignment with only 1% of the training parameters of other methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 17 of its 18 references that are in the map. First posted on arXiv 7 Mar 2024.

Builds on

17 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 733
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  12. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
4 more
  1. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  2. 2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
  3. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimizationcited by 244
  4. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74

Built on by

Cited by year

Citing papers per year: 2024: 2.

2 papers.

  1. 2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
  2. 2024On the Transformations across Reward Model, Parameter Update, and In-Context Promptcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  2. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback
  3. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  4. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  5. 2024Investigating on RLHF methodology
  6. 2024On The Global Convergence Of Online RLHF With Neural Parametrization
  7. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment
  8. 2023Aligning Large Language Models through Synthetic Feedback