mlsanityData to 7 Oct 2026

On a Connection Between Imitation Learning and RLHF

arXiv 2503.05079 ↗First posted 7 Mar 2025cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

This work studies the alignment of large language models with preference data from an imitation learning perspective. We establish a close theoretical connection between reinforcement learning from human feedback RLHF and imitation learning (IL), revealing that RLHF implicitly performs imitation learning on the preference data distribution. Building on this connection, we propose DIL, a principled framework that directly optimizes the imitation learning objective. DIL provides a unified imitation learning perspective on alignment, encompassing existing alignment algorithms as special cases while naturally introducing new variants. By bridging IL and RLHF, DIL offers new insights into alignment with RLHF. Extensive experiments demonstrate that DIL outperforms existing methods on various challenging benchmarks.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 44 of its 63 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

44 of 63 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameters2 Feb
  2. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment19 Dec
  3. 2024How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective14 Oct
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  5. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  6. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  7. 2024Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation16 Jan
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  10. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 34

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2022Generative Adversarial Networkscited by 5,603
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2020Learning to summarize from human feedbackcited by 2,562
  12. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
22 more
  1. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2021How to Train Your Energy-Based Modelscited by 271
  7. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  8. 2023Nash Learning from Human Feedbackcited by 220
  9. 2023Fine-tuning Language Models for Factualitycited by 219
  10. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoningcited by 195
  11. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  12. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  13. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  14. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  15. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  16. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  17. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  18. 2023Contrastive Difference Predictive Codingcited by 35
  19. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learningcited by 27
  20. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  21. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  22. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 4.

13 papers.

Sort:

By area:

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2026Learning beyond Teacher: Generalized On-Policy Distillation with Reward Extrapolationcited by 157
  3. 2025Explain Before You Answer: A Survey on Compositional Visual Reasoningcited by 26
  4. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuningcited by 13
  5. 2025Doubly Robust Alignment for Large Language Modelscited by 11
  6. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  7. 2025Inference-time Alignment in Continuous Spacecited by 7
  8. 2025The Hidden Link Between RLHF and Contrastive Learningcited by 5
  9. 2025Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMscited by 2
  10. 2026Reward Prediction with Factorized World Statescited by 2
  11. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
  12. 2026Towards Disentangled Preference Optimization Dynamics: Suppress the Loser, Preserve the Winnercited by 1
1 more
  1. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Beyond RLHF: A Unified Theoretical Framework of Alignment
  2. 2025Clone-Robust AI Alignment
  3. 2025RLHF: A comprehensive Survey for Cultural, Multimodal and Low Latency Alignment Methods
  4. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  5. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint
  6. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  7. 2026From RLHF to Direct Alignment: A Theoretical Unification of Preference Learning for Large Language Models
  8. 2024Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs