mlsanityData to 7 Oct 2026

DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scales

arXiv 2308.01320 ↗First posted 2 Aug 2023cs.LG cs.AI cs.CLcited by 107

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

ChatGPT-like models have revolutionized various applications in artificial intelligence, from summarization and coding to translation, matching or even surpassing human performance. However, the current landscape lacks an accessible, efficient, and cost-effective end-to-end RLHF (Reinforcement Learning with Human Feedback) training pipeline for these powerful models, particularly when training at the scale of billions of parameters. This paper introduces DeepSpeed-Chat, a novel system that democratizes RLHF training, making it accessible to the AI community. DeepSpeed-Chat offers three key capabilities: an easy-to-use training and inference experience for ChatGPT-like models, a DeepSpeed-RLHF pipeline that replicates the training pipeline from InstructGPT, and a robust DeepSpeed-RLHF system that combines various optimizations for training and inference in a unified way. The system delivers unparalleled efficiency and scalability, enabling training of models with hundreds of billions of parameters in record time and at a fraction of the cost. With this development, DeepSpeed-Chat paves the way for broader access to advanced RLHF training, even for data scientists with limited resources, thereby fostering innovation and further development in the field of AI.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 8 of its 13 references that are in the map. First posted on arXiv 2 Aug 2023.

Builds on

8 of 13 references.

References · 8

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2020Learning to summarize from human feedbackcited by 2,562
  8. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 15; 2024: 37; 2025: 28; 2026: 27.

107 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  3. 2023ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Modelscited by 184
  4. 2023Efficient Large Language Models: A Surveycited by 183
  5. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  6. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 133
  7. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  8. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 99
  9. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  10. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  11. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularizationcited by 72
  12. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkcited by 70
38 more
  1. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 65
  2. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  3. 2024EmoLLMs: A Series of Emotional Large Language Models and Annotation Tools for Comprehensive Affective Analysiscited by 62
  4. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensemblescited by 55
  5. 2024Developing ChemDFM as a large language foundation model for chemistrycited by 51
  6. 2025RLinf: Flexible and Efficient Large-scale Reinforcement Learning via Macro-to-Micro Flow Transformationcited by 51
  7. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  8. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  9. 2023Efficient RLHF: Reducing the Memory Usage of PPOcited by 40
  10. 2024NeMo-Aligner: Scalable Toolkit for Efficient Model Alignmentcited by 40
  11. 2023PRCA: Fitting Black-Box Large Language Models for Retrieval Question Answering via Pluggable Reward-Driven Contextual Adaptercited by 38
  12. 2024Towards Efficient Exact Optimization of Language Model Alignmentcited by 37
  13. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  14. 2025AsyncFlow: An Asynchronous Streaming RL Framework for Efficient LLM Post-Trainingcited by 37
  15. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  16. 2025Laminar: A Scalable Asynchronous RL Post-Training Frameworkcited by 35
  17. 2024ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocationcited by 34
  18. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecyclecited by 34
  19. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  20. 2024ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedbackcited by 25
  21. 2023Carve3D: Improving Multi-view Reconstruction Consistency for Diffusion Models with RL Finetuningcited by 22
  22. 2024Solving General Natural-Language-Description Optimization Problems with Large Language Modelscited by 22
  23. 2025Seer: Online Context Learning for Fast Synchronous LLM Reinforcement Learningcited by 21
  24. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Settingcited by 20
  25. 2024PokerGPT: An End-to-End Lightweight Solver for Multi-Player Texas Hold'em via Large Language Modelcited by 20
  26. 2024Dual Active Learning for Reinforcement Learning from Human Feedbackcited by 19
  27. 2026SOP: A Scalable Online Post-Training System for Vision-Language-Action Modelscited by 19
  28. 2025Alignment for Efficient Tool Calling of Large Language Modelscited by 17
  29. 2024HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedbackcited by 15
  30. 2024MA-RLHF: Reinforcement Learning from Human Feedback with Macro Actionscited by 15
  31. 2023Aligning Language Models with Offline Learning from Human Feedbackcited by 14
  32. 2024SmileyLlama: Modifying Large Language Models for Directed Chemical Space Explorationcited by 14
  33. 2025Curiosity-Driven Reinforcement Learning from Human Feedbackcited by 14
  34. 2025ReSpec: Towards Optimizing Speculative Decoding in Reinforcement Learning Systemscited by 14
  35. 2023An Adaptive Placement and Parallelism Framework for Accelerating RLHF Trainingcited by 13
  36. 2025Enhancing LLM Reliability via Explicit Knowledge Boundary Modelingcited by 13
  37. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  38. 2026DORA: A Scalable Asynchronous Reinforcement Learning System for Language Model Trainingcited by 13

50 most cited of 107 All 107 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024RLHF Workflow: From Reward Modeling to Online RLHF
  2. 2026GrowMTP: Can RL Grow Its Own Draft Head?
  3. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale
  4. 2022DeepSpeed Data Efficiency: Improving Deep Learning Model Quality and Training Efficiency via Efficient Data Sampling and Routing
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  6. 2025A Review of DeepSeek Models' Key Innovative Techniques
  7. 2025A Showdown of ChatGPT vs DeepSeek in Solving Programming Tasks
  8. 2023Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models