mlsanityData to 14 Oct 2026

Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning

arXiv 2405.15624 ↗First posted 24 May 2024cs.LG cs.AIcited by 27

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

also in 1 more
Abstract (arXiv)

Aligning Large Language Models (LLMs) is crucial for enhancing their safety and utility. However, existing methods, primarily based on preference datasets, face challenges such as noisy labels, high annotation costs, and privacy concerns. In this work, we introduce Alignment from Demonstrations (AfD), a novel approach leveraging high-quality demonstration data to overcome these challenges. We formalize AfD within a sequential decision-making framework, highlighting its unique challenge of missing reward signals. Drawing insights from forward and inverse reinforcement learning, we introduce divergence minimization objectives for AfD. Analytically, we elucidate the mass-covering and mode-seeking behaviors of various approaches, explaining when and why certain methods are superior. Practically, we propose a computationally efficient algorithm that extrapolates over a tailored reward model for AfD. We validate our key insights through experiments on the Harmless and Helpful tasks, demonstrating their strong empirical performance while maintaining simplicity.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 65 of its 91 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

65 of 91 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimation8 Mar
  2. 2024Dense Reward for Free in Reinforcement Learning from Human Feedback1 Feb
  3. 2023Nash Learning from Human Feedback1 Dec
  4. 2023Contrastive Preference Learning: Learning from Human Feedback without RL20 Oct
  5. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversity10 Oct
  6. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  7. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
  8. 2023Secrets of RLHF in Large Language Models Part I: PPO11 Jul
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  12. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
2 more
  1. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  2. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 51

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  7. 2016Improved Techniques for Training GANscited by 2,970
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2016Generative Adversarial Imitation Learningcited by 1,597
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
38 more
  1. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  3. 2016Deep Exploration via Bootstrapped DQNcited by 527
  4. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 506
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  6. 2020Towards a Human-like Open-Domain Chatbotcited by 470
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 376
  9. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  10. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  11. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 260
  12. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  13. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  14. 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 219
  15. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 204
  16. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 190
  17. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
  18. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraintscited by 174
  19. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  20. 2020A Distributional Approach to Controlled Text Generationcited by 119
  21. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  22. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  23. 2018Language GANs Falling Shortcited by 97
  24. 2022On Decoding Strategies for Neural Text Generatorscited by 90
  25. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  26. 2023Data Quality in Imitation Learningcited by 75
  27. 2021What Matters for Adversarial Imitation Learning?cited by 70
  28. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54
  29. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
  30. 2017Maximum-Likelihood Augmented Discrete Generative Adversarial Networkscited by 39
  31. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generationcited by 28
  32. 2023Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Gamecited by 27
  33. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  34. 2020Why Spectral Normalization Stabilizes GANs: Analysis and Improvementscited by 21
  35. 2022Improving GANs with A Dynamic Discriminatorcited by 20
  36. 2020Self-Adversarial Learning with Comparative Discrimination for Text Generationcited by 11
  37. 2019ARAML: A Stable Adversarial Training Framework for Text Generationcited by 9
  38. 2024Retrieval Augmented Thought Process for Private Data Handling in Healthcarecited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 14; 2026: 4.

27 papers.

Sort:

By area:

  1. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  2. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  3. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 70
  4. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  5. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  6. 2025Step-wise Adaptive Integration of Supervised Fine-tuning and Reinforcement Learning for Task-Specific LLMscited by 25
  7. 2025Diverse Preference Learning for Capabilities and Alignmentcited by 25
  8. 2024SPRec: Self-Play to Debias LLM-based Recommendationcited by 18
  9. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  10. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  11. 2024Forward KL Regularized Preference Optimization for Aligning Diffusion Policiescited by 10
  12. 2024ProgressGym: Alignment with a Millennium of Moral Progresscited by 9
15 more
  1. 2025Evaluating and Aligning Human Economic Risk Preferences in LLMscited by 8
  2. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  3. 2025Improving Diversity in Language Models: When Temperature Fails, Change the Losscited by 7
  4. 2025Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrationscited by 6
  5. 2024LRHP: Learning Representations for Human Preferences via Preference Pairscited by 5
  6. 2025OpenReview Should be Protected and Leveraged as a Community Asset for Research in the Era of Large Language Modelscited by 5
  7. 2025Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directionscited by 5
  8. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignmentcited by 4
  9. 2025Learning Natural Language Constraints for Safe Reinforcement Learning of Language Agentscited by 2
  10. 2026SPS: Steering Probability Squeezing for Better Exploration in Reinforcement Learning for Large Language Modelscited by 2
  11. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Onlycited by 1
  12. 2026rePIRL: Learn PRM with Inverse RL for LLM Reasoningcited by 1
  13. 2026Equalized Generative Treatment: Matching f-divergences for Fairness in Generative Modelscited by 1
  14. 2025Where You Go is Who You Are: Behavioral Theory-Guided LLMs for Inverse Reinforcement Learningcited by 0
  15. 2026Turn-level Multiscale Density Ratio Estimation for LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  2. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
  3. 2026Inverse RL Helps Align AI by Imitating Humans
  4. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  5. 2025Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment
  6. 2024Imitating Language via Scalable Inverse Reinforcement Learning
  7. 2025AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation
  8. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning