mlsanityData to 7 Oct 2026

How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective

arXiv 2410.10093 ↗First posted 14 Oct 2024cs.CL cs.LGcited by 14

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

also in 1 more
Abstract (arXiv)

This paper introduces a novel generalized self-imitation learning (GSIL) framework, which effectively and efficiently aligns large language models with offline demonstration data. We develop GSIL by deriving a surrogate objective of imitation learning with density ratio estimates, facilitating the use of self-generated data and optimizing the imitation learning objective with simple classification losses. GSIL eliminates the need for complex adversarial training in standard imitation learning, achieving lightweight and efficient fine-tuning for large language models. In addition, GSIL encompasses a family of offline losses parameterized by a general class of convex functions for density ratio estimation and enables a unified view for alignment with demonstration data. Extensive experiments show that GSIL consistently and significantly outperforms baselines in many challenging benchmarks, such as coding (HuamnEval), mathematical reasoning (GSM8K) and instruction-following benchmark (MT-Bench).

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 57 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

35 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive20 Feb
  2. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignment8 Feb
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  4. 2023Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints28 Sep
  5. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  6. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 29

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2016Generative Adversarial Imitation Learningcited by 1,576
17 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  2. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  3. 2018An Algorithmic Perspective on Imitation Learningcited by 455
  4. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 400
  5. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  6. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  7. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  8. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  9. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
  10. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 139
  11. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  12. 2019Learning to Generalize from Sparse and Underspecified Rewardscited by 39
  13. 2023Simple and Asymmetric Graph Contrastive Learning without Augmentationscited by 37
  14. 2023Learning How to Propagate Messages in Graph Neural Networkscited by 23
  15. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18
  16. 2016Improving Policy Gradient by Exploring Under-appreciated Rewardscited by 14
  17. 2022Towards Generalizable Graph Contrastive Learning: An Information Theory Perspectivecited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 10; 2026: 3.

14 papers.

Sort:

By area:

  1. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  2. 2025SimPER: A Minimalist Approach to Preference Alignment without Hyperparameterscited by 25
  3. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  4. 2025Rethinking Diverse Human Preference Learning through Principal Component Analysiscited by 10
  5. 2025Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learningcited by 10
  6. 2025Incentivizing Strong Reasoning from Weak Supervisioncited by 8
  7. 2025InfoPO: On Mutual Information Maximization for Large Language Model Alignmentcited by 7
  8. 2025Inference-time Alignment in Continuous Spacecited by 7
  9. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  10. 2025SynPO: Synergizing Descriptiveness and Preference Optimization for Video Detailed Captioningcited by 3
  11. 2025Beyond Imitation: Recovering Dense Rewards from Demonstrationscited by 1
  12. 2026Match or Replay: Self Imitating Proximal Policy Optimizationcited by 0
2 more
  1. 2026Learn from Your Mistakes: Tree-like Self-Play for Secure Code LLMscited by 0
  2. 2026Turn-level Multiscale Density Ratio Estimation for LLM Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning
  2. 2024Supervised Fine-Tuning as Inverse Reinforcement Learning
  3. 2024Aligning Language Models with Demonstrated Feedback
  4. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models
  5. 2024Imitating Language via Scalable Inverse Reinforcement Learning
  6. 2025Aligning Large Language Models via Fully Self-Synthetic Data
  7. 2025Towards Efficient and Effective Alignment of Large Language Models
  8. 2021A General Language Assistant as a Laboratory for Alignment