mlsanityData to 14 Oct 2026

Soft Policy Optimization: Online Off-Policy RL for Sequence Models

arXiv 2503.05453 ↗First posted 7 Mar 2025cs.LG cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

RL-based post-training of language models is almost exclusively done using on-policy methods such as PPO. These methods cannot learn from arbitrary sequences such as those produced earlier in training, in earlier runs, by human experts or other policies, or by decoding and exploration methods. This results in severe sample inefficiency and exploration difficulties, as well as a potential loss of diversity in the policy responses. Moreover, asynchronous PPO implementations require frequent and costly model transfers, and typically use value models which require a large amount of memory. In this paper we introduce Soft Policy Optimization (SPO), a simple, scalable and principled Soft RL method for sequence model policies that can learn from arbitrary online and offline trajectories and does not require a separate value model. In experiments on code contests, we shows that SPO outperforms PPO on pass@10, is significantly faster and more memory efficient, is able to benefit from off-policy data, enjoys improved stability, and learns more diverse (i.e. soft) policies.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 26 of its 30 references that are in the map. First posted on arXiv 7 Mar 2025.

Builds on

26 of 30 references.

References · 26

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  6. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  9. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  10. 2025Process Reinforcement through Implicit Rewardscited by 504
  11. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 482
  12. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 290
14 more
  1. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  2. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  3. 2017Equivalence Between Policy Gradients and Soft Q-Learningcited by 173
  4. 2024Free Process Rewards without Process Labelscited by 165
  5. 2023TACO: Topics in Algorithmic COde generation datasetcited by 160
  6. 2024RLEF: Grounding Code LLMs in Execution Feedback with Reinforcement Learningcited by 151
  7. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
  8. 2016Sequence Tutor: Conservative Fine-Tuning of Sequence Generation Models with KL-controlcited by 101
  9. 2024Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Modelscited by 70
  10. 2024Offline Regularised Reinforcement Learning for Large Language Models Alignmentcited by 48
  11. 2013Credal Networks under Maximum Entropycited by 13
  12. 2023VA-learning as a more efficient alternative to Q-learningcited by 11
  13. 2024What type of inference is planning?cited by 9
  14. 2022Calculus on MDPs: Potential Shaping as a Gradientcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 4.

12 papers.

  1. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
  2. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Modelscited by 93
  3. 2025ExGRPO: Learning to Reason from Experiencecited by 53
  4. 2025SPG: Sandwiched Policy Gradient for Masked Diffusion Language Modelscited by 41
  5. 2025On a few pitfalls in KL divergence gradient estimation for RLcited by 29
  6. 2025Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Modelcited by 14
  7. 2025A-3PO: Accelerating Asynchronous LLM Training with Staleness-aware Proximal Policy Approximationcited by 12
  8. 2026Generate, Filter, Control, Replay: A Comprehensive Survey of Rollout Strategies for LLM Reinforcement Learningcited by 9
  9. 2025The Era of Agentic Organization: Learning to Organize with Language Modelscited by 7
  10. 2026V0: A Generalist Value Model for Any Policy at State Zerocited by 7
  11. 2026Mask-Aware Policy Gradients for Diffusion Language Modelscited by 3
  12. 2026DecompRL: Solving Harder Problems by Learning Modular Code Generationcited by 0