mlsanityData to 14 Oct 2026

Bridging Offline and Online Reinforcement Learning for LLMs

arXiv 2506.21495 ↗First posted 26 Jun 2025cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

We investigate the effectiveness of reinforcement learning methods for finetuning large language models when transitioning from offline to semi-online to fully online regimes for both verifiable and non-verifiable tasks. Our experiments cover training on verifiable math as well as non-verifiable instruction following with a set of benchmark evaluations for both. Across these settings, we extensively compare online and semi-online Direct Preference Optimization and Group Reward Policy Optimization objectives, and surprisingly find similar performance and convergence between these variants, which all strongly outperform offline methods. We provide a detailed analysis of the training dynamics and hyperparameter selection strategies to achieve optimal results. Finally, we show that multi-tasking with verifiable and non-verifiable rewards jointly yields improved performance across both task types.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 39 of its 47 references that are in the map. First posted on arXiv 26 Jun 2025.

Builds on

39 of 47 references.

References · 39

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  9. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  10. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  11. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  12. 2017Deep reinforcement learning from human preferencescited by 3,912
27 more
  1. 2023Let's Verify Step by Stepcited by 3,789
  2. 2015Trust Region Policy Optimizationcited by 2,580
  3. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  4. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  5. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  6. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  7. 2024Self-Rewarding Language Modelscited by 681
  8. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  9. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  10. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 557
  11. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  12. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  13. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  14. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  15. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 323
  16. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 291
  17. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  18. 2024Direct Language Model Alignment from Online AI Feedbackcited by 260
  19. 2024Iterative Reasoning Preference Optimizationcited by 246
  20. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  21. 2024Disentangling Length from Quality in Direct Preference Optimizationcited by 196
  22. 2023Self-Alignment with Instruction Backtranslationcited by 151
  23. 2023Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Losscited by 108
  24. 2024How to Evaluate Reward Models for RLHFcited by 88
  25. 2024Preference Learning Algorithms Do Not Learn Preference Rankingscited by 48
  26. 2024Thinking LLMs: General Instruction Following with Thought Generationcited by 45
  27. 2024BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignmentcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 16.

25 papers.

Sort:

By area:

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
  2. 2025Jointly Reinforcing Diversity and Quality in Language Model Generationscited by 80
  3. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 32
  4. 2025RL Fine-Tuning Heals OOD Forgetting in SFTcited by 31
  5. 2025Learning to Reason for Factualitycited by 24
  6. 2025Language Models Can Learn from Verbal Feedback Without Scalar Rewardscited by 18
  7. 2025Beyond Correctness: Rewarding Faithful Reasoning in Retrieval-Augmented Generationcited by 6
  8. 2026Self-Improving Pretraining: using post-trained models to pretrain better modelscited by 6
  9. 2026ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuningcited by 6
  10. 2026LeVo 2: Stable and Melodious Song Generation via Hierarchical Representation Modeling and Progressive Post-Trainingcited by 6
  11. 2025SERL: Self-Examining Reinforcement Learning on Open-Domaincited by 5
  12. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
13 more
  1. 2026Property-driven Protein Inverse Folding With Multi-Objective Preference Alignmentcited by 3
  2. 2026Reasoning over mathematical objects: on-policy reward modeling and test time aggregationcited by 3
  3. 2026Multi2: Hierarchical Multi-Agent Decision-Making with LLM-Based Agents in Interactive Environmentscited by 2
  4. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  5. 2026PCL-Reasoner-V1.5: Advancing Math Reasoning with Offline Reinforcement Learningcited by 1
  6. 2026Thinking Seeds: Leveraging Historical Diversity for Position-Aware RL in LLMscited by 1
  7. 2026ReNCE: Learning to Reason by Noise Contrastive Estimationcited by 1
  8. 2026Smaller Models are Natural Explorers for Policy-Level Diversity in GRPOcited by 1
  9. 2026ConRub-Med: Reinforcement Learning with Consensus Rubrics for Open-Ended Medical Question Answeringcited by 1
  10. 2025Humanline: Online Alignment as Perceptual Losscited by 0
  11. 2026PaperGuide: Making Small Language-Model Paper-Reading Agents More Efficientcited by 0
  12. 2026Step-by-Step Optimization-like Reasoning in LLMs over Expanding Search Spacescited by 0
  13. 2026Inducing Emergent Misalignment from Reward Hacks with Iterative DPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective
  2. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  3. 2025Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training
  4. 2025Reinforcement Learning Meets Large Language Models: A Survey of Advancements and Applications Across the LLM Lifecycle
  5. 2024Reinforcement Learning for LLM Post-Training: A Survey
  6. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  7. 2026UpSkill: Mutual Information Skill Learning for Structured Response Diversity in LLMs
  8. 2026Likelihood-Based Reward Designs for General LLM Reasoning