mlsanityData to 7 Oct 2026

A Note on Hybrid Online Reinforcement and Imitation Learning for LLMs: Formulations and Algorithms

arXiv 2512.23097 ↗First posted 28 Dec 2025cs.LG cs.AI cs.CLcited by 2

On the map: Offline RL and imitation learning from data → Language-conditioned policies and VLAs

Abstract (arXiv)

We present a unified framework for Large Language Model (LLM) fine-tuning that integrates Imitation Learning and Reinforcement Learning. By analyzing the gradient of a composite objective combining trajectory-level KL divergence with task rewards, we derive a natural decomposition into two components: (1) an analytically computable Dense Gradient for token-level imitation, and (2) a Monte Carlo estimated Sparse Gradient for long-horizon reward optimization. The Dense Gradient admits a closed-form logit-level formula, enabling efficient GPU implementation.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data. Builds on 3 of its 10 references that are in the map. First posted on arXiv 28 Dec 2025.

Builds on

3 of 10 references.

References · 3

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Multi-Rollout On-Policy Distillation via Peer Successes and Failurescited by 8
  2. 2026A First-Principles Derivation of LLM Policy Optimization: From Expected Reward to GRPO and Its Structural Extensionscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  2. 2025Bridging Offline and Online Reinforcement Learning for LLMs
  3. 2024Imitating Language via Scalable Inverse Reinforcement Learning
  4. 2025Stabilizing Reinforcement Learning with LLMs: Formulation and Practices
  5. 2025PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
  6. 2018Accelerating Imitation Learning with Predictive Models
  7. 2025Improving Retrospective Language Agents via Joint Policy Gradient Optimization
  8. 2024Countering Reward Over-optimization in LLM with Demonstration-Guided Reinforcement Learning