mlsanityData to 7 Oct 2026

RL-finetuning LLMs from on- and off-policy data with a single algorithm

arXiv 2503.19612 ↗First posted 25 Mar 2025cs.LG stat.MLcited by 11

Not on the map. Why?

Abstract (arXiv)

We introduce a novel reinforcement learning algorithm (AGRO, for Any-Generation Reward Optimization) for fine-tuning large-language models. AGRO leverages the concept of generation consistency, which states that the optimal policy satisfies the notion of consistency across any possible generation of the model. We derive algorithms that find optimal solutions via the sample-based policy gradient and provide theoretical guarantees on their convergence. Our experiments demonstrate the effectiveness of AGRO in both on-policy and off-policy settings, showing improved performance on the mathematical reasoning dataset over baseline algorithms.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 33 of its 45 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

33 of 45 references.

References · 33

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
21 more
  1. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  2. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  3. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  4. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  5. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  6. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  7. 2021GFlowNet Foundationscited by 286
  8. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  9. 2024Direct Language Model Alignment from Online AI Feedbackcited by 257
  10. 2023Nash Learning from Human Feedbackcited by 220
  11. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  12. 2017Bridging the Gap Between Value and Policy Based Reinforcement Learningcited by 178
  13. 2024Generalized Preference Optimization: A Unified Approach to Offline Alignmentcited by 155
  14. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 146
  15. 2024Understanding the performance gap between online and offline alignment algorithmscited by 110
  16. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  17. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  18. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  19. 2023Maximum entropy GFlowNets with soft Q-learningcited by 30
  20. 2024Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashioncited by 13
  21. 2024Averaging log-likelihoods in direct alignmentcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 5.

11 papers.

  1. 2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 88
  2. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  3. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  4. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 27
  5. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  6. 2026LLMs Can Learn to Reason Via Off-Policy RLcited by 10
  7. 2026Efficient RL Training for LLMs with Experience Replaycited by 8
  8. 2025ShiQ: Bringing back Bellman to LLMscited by 3
  9. 2026f-Trajectory Balance: A Loss Family for Tuning GFlowNets, Generative Models, and LLMs with Off- and On-Policy Datacited by 3
  10. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  11. 2026Reinforcement Learning for Compositional Generalization with Outcome-Level Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMs
  2. 2025Accelerating RL for LLM Reasoning with Optimal Advantage Regression
  3. 2025Token-Efficient RL for LLM Reasoning
  4. 2025One-Token Rollout: Guiding Supervised Fine-Tuning of LLMs with Policy Gradient
  5. 2022Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
  6. 2026ARMOR: Stabilizing On-Policy LLM RL with Off-Policy Anchor Samples
  7. 2024RLSF: Fine-tuning LLMs via Symbolic Feedback
  8. 2025A Technical Survey of Reinforcement Learning Techniques for Large Language Models