mlsanityData to 7 Oct 2026

ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

arXiv 2404.00934 ↗First posted 1 Apr 2024cs.CLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

ChatGLM is a free-to-use AI service powered by the ChatGLM family of large language models (LLMs). In this paper, we present the ChatGLM-RLHF pipeline -- a reinforcement learning from human feedback (RLHF) system -- designed to enhance ChatGLM's alignment with human preferences. ChatGLM-RLHF encompasses three major components: the collection of human preference data, the training of the reward model, and the optimization of policies. Throughout the process of integrating ChatGLM-RLHF into production, we encountered and addressed several unprecedented challenges. We introduce the strategies to mitigate reward variance for stabilized large-scale training, implement model parallelism with fused gradient-descent, and design regularization constraints to avoid catastrophic forgetting in LLMs. Experiments show that ChatGLM-RLHF brings significant improvements in alignment tasks compared to the supervised fine-tuned (SFT) version of ChatGLM. For instance, it achieves on average 15% more wins against ChatGLM-SFT in Chinese alignment tasks. The work presents our practices of aligning LLMs with human preferences, offering insights into the challenges and solutions in RLHF implementations.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 37 references that are in the map. First posted on arXiv 1 Apr 2024.

Builds on

30 of 37 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023SALMON: Self-Alignment with Instructable Reward Models9 Oct
  2. 2023Preference Ranking Optimization for Human Alignment30 Jun
  3. 2023LIMA: Less Is More for Alignment18 May
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
14 more
  1. 2023Qwen Technical Reportcited by 3,273
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  4. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  7. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  8. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  9. 2023Secrets of RLHF in Large Language Models Part I: PPOcited by 226
  10. 2021Alignment of Language Agentscited by 167
  11. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107
  12. 2023Loose lips sink ships: Mitigating Length Bias in Reinforcement Learning from Human Feedbackcited by 78
  13. 2023Pairwise Proximal Policy Optimization: Harnessing Relative Feedback for LLM Alignmentcited by 57
  14. 2023Revisiting Discriminative vs. Generative Classifiers: Theory and Implicationscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 9; 2026: 2.

25 papers.

Sort:

By area:

  1. 2024LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMscited by 142
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  4. 2024LongReward: Improving Long-context Large Language Models with AI Feedbackcited by 34
  5. 2024Beyond Scalar Reward Model: Learning Generative Judge from Preference Datacited by 30
  6. 2024Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergencecited by 29
  7. 2024Post-hoc Reward Calibration: A Case Study on Length Biascited by 29
  8. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
  9. 20243D-Properties: Identifying Challenges in DPO and Charting a Path Forwardcited by 24
  10. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  11. 2024SPaR: Self-Play with Tree-Search Refinement to Improve Instruction-Following in Large Language Modelscited by 23
  12. 2024Planning in the Dark: LLM-Symbolic Planning Pipeline without Expertscited by 15
13 more
  1. 2025LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learningcited by 13
  2. 2024Tele-FLM Technical Reportcited by 12
  3. 2024Better RAG using Relevant Information Gaincited by 7
  4. 2025Reference-Free Rating of LLM Responses via Latent Informationcited by 4
  5. 2025IF-CRITIC: Towards a Fine-Grained LLM Critic for Instruction-Following Evaluationcited by 4
  6. 2025Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Modelscited by 3
  7. 2025SGDPO: Self-Guided Direct Preference Optimization for Language Model Alignmentcited by 3
  8. 2026ETS: Energy-Guided Test-Time Scaling for Training-Free RL Alignmentcited by 3
  9. 2025Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratchcited by 2
  10. 2024EntityCLIP: Entity-Centric Image-Text Matching via Multimodal Attentive Contrastive Learningcited by 1
  11. 2024From Words to Workflows: Automating Business Processescited by 0
  12. 2025QExplorer: Large Language Model Based Query Extraction for Toxic Content Explorationcited by 0
  13. 2026ScopeIF: Improving Scope-Aware Precise Instruction-Following in Large Language Models via Graded Reward Modelingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  2. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  3. 2024Proxy-RLHF: Decoupling Generation and Alignment in Large Language Model with Proxy
  4. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  5. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  6. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  7. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  8. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models