mlsanityData to 7 Oct 2026

Imitating Language via Scalable Inverse Reinforcement Learning

arXiv 2409.01369 ↗First posted 2 Sep 2024cs.LG cs.AI cs.CLcited by 21

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

The majority of language model training builds on imitation learning. It covers pretraining, supervised fine-tuning, and affects the starting conditions for reinforcement learning from human feedback (RLHF). The simplicity and scalability of maximum likelihood estimation (MLE) for next token prediction led to its role as predominant paradigm. However, the broader field of imitation learning can more effectively utilize the sequential structure underlying autoregressive generation. We focus on investigating the inverse reinforcement learning (IRL) perspective to imitation, extracting rewards and directly optimizing sequences instead of individual token likelihoods and evaluate its benefits for fine-tuning large language models. We provide a new angle, reformulating inverse soft-Q-learning as a temporal difference regularized extension of MLE. This creates a principled connection between MLE and IRL and allows trading off added complexity with increased performance and diversity of generations in the supervised fine-tuning (SFT) setting. We find clear advantages for IRL-based imitation, in particular for retaining diversity while maximizing task performance, rendering IRL a strong alternative on fixed SFT datasets even without online data generation. Our analysis of IRL-extracted reward functions further indicates benefits for more robust reward functions via tighter integration of supervised and preference-based LLM post-training.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 71 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

55 of 71 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning18 Mar
  2. 2020Autoregressive Knowledge Distillation through Imitation Learning15 Sep

Other references · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2020Learning to summarize from human feedbackcited by 2,562
  10. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  11. 2016Generative Adversarial Imitation Learningcited by 1,576
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
38 more
  1. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 700
  4. 2015Sequence Level Training with Recurrent Neural Networkscited by 568
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  6. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  8. 2016SeqGAN: Sequence Generative Adversarial Nets with Policy Gradientcited by 375
  9. 2019A Theory of Regularized Markov Decision Processescited by 279
  10. 2023A Survey of Reinforcement Learning from Human Feedbackcited by 259
  11. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  12. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  13. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 191
  14. 2019A Divergence Minimization Perspective on Imitation Learning Methodscited by 188
  15. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  16. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  17. 2024The pitfalls of next-token predictioncited by 156
  18. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  19. 2018Language GANs Falling Shortcited by 97
  20. 2020Error Bounds of Imitating Policies and Environmentscited by 90
  21. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  22. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  23. 2023Introduction to Latent Variable Energy-Based Models: A Path Towards Autonomous Machine Intelligencecited by 52
  24. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  25. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  26. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  27. 2016Watch This: Scalable Cost-Function Learning for Path Planning in Urban Environmentscited by 33
  28. 2024Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignmentcited by 32
  29. 2024Hybrid Inverse Reinforcement Learningcited by 31
  30. 2020TextGAIL: Generative Adversarial Imitation Learning for Text Generationcited by 28
  31. 2023LS-IQ: Implicit Reward Regularization for Inverse Reinforcement Learningcited by 28
  32. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 26
  33. 2022Proximal Point Imitation Learningcited by 25
  34. 2023Coherent Soft Imitation Learningcited by 21
  35. 2017Mutual Alignment Transfer Learningcited by 16
  36. 2023SequenceMatch: Imitation Learning for Autoregressive Sequence Modelling with Backtrackingcited by 13
  37. 2023Massively Scalable Inverse Reinforcement Learning in Google Mapscited by 11
  38. 2019ARAML: A Stable Adversarial Training Framework for Text Generationcited by 9

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 14; 2026: 6.

21 papers.

Sort:

By area:

  1. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 58
  2. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilitiescited by 36
  3. 2025Generalist Reward Models: Found Inside Large Language Modelscited by 34
  4. 2025Thought Manipulation: External Thought Can Be Efficient for Large Reasoning Modelscited by 23
  5. 2025Implicit Reward as the Bridge: A Unified View of SFT and DPO Connectionscited by 17
  6. 2024Diffusing States and Matching Scores: A New Framework for Imitation Learningcited by 16
  7. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  8. 2025Inverse Reinforcement Learning with Switching Rewards and History Dependency for Characterizing Animal Behaviorscited by 6
  9. 2025IL-SOAR : Imitation Learning with Soft Optimistic Actor cRiticcited by 5
  10. 2025From Demonstrations to Rewards: Alignment Without Explicit Human Preferencescited by 4
  11. 2025Cognitive models can reveal interpretable value trade-offs in language modelscited by 3
  12. 2025Universal Reasoner: A Single, Composable Plug-and-Play Reasoner for Frozen LLMscited by 2
9 more
  1. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Datacited by 1
  2. 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Onlycited by 1
  3. 2026Internalizing Meta-Experience into Memory for Guided Reinforcement Learning in Large Language Modelscited by 1
  4. 2026FAIL: Flow Matching Adversarial Imitation Learning for Image Generationcited by 1
  5. 2025Noise-Guided Transport for Imitation Learningcited by 0
  6. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  7. 2026Non-Adversarial Imitation Learning Provably Free of Compounding Errors: The Value Flow Mechanismcited by 0
  8. 2026Split the Differences, Pool the Rest: Provably Efficient Multi-Objective Imitationcited by 0
  9. 2026Inverse RL Helps Align AI by Imitating Humanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Inverse-Q*: Token Level Reinforcement Learning for Aligning Large Language Models Without Preference Data
  2. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  3. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities
  4. 2024Approximated Variational Bayesian Inverse Reinforcement Learning for Large Language Model Alignment
  5. 2025Language Model Distillation: A Temporal Difference Imitation Learning Perspective
  6. 2023Training Language Models with Language Feedback at Scale
  7. 2024Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning
  8. 2025GRACE: A Language Model Framework for Explainable Inverse Reinforcement Learning