mlsanityData to 7 Oct 2026

Supervised Fine-Tuning as Inverse Reinforcement Learning

arXiv 2403.12017 ↗First posted 18 Mar 2024cs.LG cs.AI cs.CLcited by 12

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

The prevailing approach to aligning Large Language Models (LLMs) typically relies on human or AI feedback and assumes access to specific types of preference datasets. In our work, we question the efficacy of such datasets and explore various scenarios where alignment with expert demonstrations proves more realistic. We build a sequential decision-making framework to formulate the problem of aligning LLMs using demonstration datasets. Drawing insights from inverse reinforcement learning and imitation learning, we introduce various approaches for divergence minimization in the LLM alignment tasks. Our analysis highlights the mass-covering and mode-seeking behaviors of these different approaches. Inclusively, we examine the pros and cons of the classical supervised fine-tuning method, elaborating on scenarios where different methods shine.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 49 references that are in the map. First posted on arXiv 18 Mar 2024.

Builds on

30 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Direct Language Model Alignment from Online AI Feedback7 Feb
  2. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  3. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  4. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 26

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2017Deep reinforcement learning from human preferencescited by 3,846
  4. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  7. 2016Generative Adversarial Imitation Learningcited by 1,576
  8. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  11. 2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
  12. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
14 more
  1. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  2. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  3. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
  4. 2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
  5. 2023Nash Learning from Human Feedbackcited by 220
  6. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  7. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  8. 2021Urban Driver: Learning to Drive from Real-world Demonstrations Using Policy Gradientscited by 122
  9. 2020A Distributional Approach to Controlled Text Generationcited by 117
  10. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  11. 2022On Decoding Strategies for Neural Text Generatorscited by 89
  12. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  13. 2021What Matters for Adversarial Imitation Learning?cited by 70
  14. 2020Strictly Batch Imitation Learning by Energy-based Distribution Matchingcited by 54

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 9.

12 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  2. 2025GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clippingcited by 48
  3. 2024Margin-aware Preference Optimization for Aligning Diffusion Models without Referencecited by 45
  4. 2024Imitating Language via Scalable Inverse Reinforcement Learningcited by 21
  5. 2025Fine-Grained GRPO for Precise Preference Alignment in Flow Modelscited by 21
  6. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  7. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  8. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  9. 2025Agent-Based Uncertainty Awareness Improves Automated Radiology Report Labeling with an Open-Source Large Language Modelcited by 1
  10. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
  11. 2024Building an Ethical and Trustworthy Biomedical AI Ecosystem for the Translational and Clinical Integration of Foundational Modelscited by 0
  12. 2025Noise-Guided Transport for Imitation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  2. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  3. 2023Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment
  4. 2024Aligning Large Language Models via Fine-grained Supervision
  5. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
  6. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning
  7. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
  8. 2026Alignment Dynamics in LLM Fine-Tuning