mlsanityData to 14 Oct 2026

Aligning Language Models with Offline Learning from Human Feedback

arXiv 2308.12050 ↗First posted 23 Aug 2023cs.CL cs.AIcited by 14

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Learning from human preferences is crucial for language models (LMs) to effectively cater to human needs and societal values. Previous research has made notable progress by leveraging human feedback to follow instructions. However, these approaches rely primarily on online learning techniques like Proximal Policy Optimization (PPO), which have been proven unstable and challenging to tune for language models. Moreover, PPO requires complex distributed system implementation, hindering the efficiency of large-scale distributed training. In this study, we propose an offline learning from human feedback framework to align LMs without interacting with environments. Specifically, we explore filtering alignment (FA), reward-weighted regression (RWR), and conditional alignment (CA) to align language models to human preferences. By employing a loss function similar to supervised fine-tuning, our methods ensure more stable model training than PPO with a simple machine learning system~(MLSys) and much fewer (around 9%) computing resources. Experimental results demonstrate that conditional alignment outperforms other offline alignment methods and is comparable to PPO.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 19 of its 30 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

19 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignment14 Apr
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  6. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  7. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  9. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2020GLU Variants Improve Transformercited by 1,804
  12. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 108
4 more
  1. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  2. 2021Rethinking the Implementation Tricks and Monotonicity Constraint in Cooperative Multi-Agent Reinforcement Learningcited by 68
  3. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  4. 2018Where Did My Optimum Go?: An Empirical Analysis of Gradient Descent Optimization in Policy Gradient Methodscited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 6; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 55
  4. 2024A Survey on Human Preference Learning for Large Language Modelscited by 24
  5. 2025Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approachcited by 24
  6. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Modelscited by 21
  7. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  8. 2025Multi-Objective Preference Optimization: Improving Human Alignment of Generative Modelscited by 13
  9. 2024Towards Aligning Language Models with Textual Feedbackcited by 5
  10. 2025OPPO: Accelerating PPO-based RLHF via Pipeline Overlapcited by 3
  11. 2025Multi-Objective Reward and Preference Optimization: Theory and Algorithmscited by 0
  12. 2026Supervised Fine-Tuning Needs to Unlock the Potential of Token Prioritycited by 0
2 more
  1. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuningcited by 0
  2. 2026MADA-RL: Multi-Agent Debate-Aware Reinforcement Learning for Parameter-Efficient Reasoning in Compact Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024SAIL: Self-Improving Efficient Online Alignment of Large Language Models
  2. 2024Understanding the Learning Dynamics of Alignment with Human Feedback
  3. 2024Unified Preference Optimization: Language Model Alignment Beyond the Preference Frontier
  4. 2023ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference
  5. 2024Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback
  6. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  7. 2023Data-Efficient Alignment of Large Language Models with Human Feedback Through Natural Language
  8. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models