mlsanityData to 7 Oct 2026

FBOS-RL: Feedback-Driven Bi-Objective Synergistic Reinforcement Learning

arXiv 2605.20256 ↗First posted 18 May 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning has become a cornerstone for aligning and unlocking the reasoning capabilities of large-scale models. At its core, the training loop of GRPO and its variants alternates between rollout sampling and policy update: the policy first samples rollouts from its action space, and then updates its parameters according to the advantages computed over them. Unlike supervised learning, where each gradient step is anchored to an explicit ground-truth target, the optimal gradient direction for updating model parameters in this setting is not known a priori; the high-quality rollouts drawn during the sampling stage therefore act as the implicit "teacher" that guides every parameter update. However, mainstream RL algorithms such as GRPO adopt a simple sampling scheme that conditions all rollouts on the same original prompt. When a task lies beyond the policy model's current capability, this sampling scheme rarely yields a high-quality rollout, leaving the policy model without a meaningful gradient direction when updating its parameters, which causes training to stall. To address this issue, we propose FBOS-RL. Specifically, we let the model perform Feedback-Guided Exploration Enhancement based on the feedback provided by the environment, and on top of this we design two mutually reinforcing training objectives: EPA and ECC. Extensive experiments demonstrate that EPA and ECC can mutually reinforce each other, forming a positive flywheel effect that significantly improves both the training efficiency and the final performance ceiling of reinforcement learning. Specifically, under both an identical number of rollouts and the same number of training steps, FBOS-RL learns substantially faster than GRPO and feedback-based baselines and ultimately attains a higher performance ceiling, while exhibiting higher policy entropy and lower gradient norms throughout training.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 34 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

29 of 34 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
  2. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  3. 2025Reinforcement Learning for Reasoning in Large Language Models with One Training Example29 Apr
  4. 2025Learning to Reason under Off-Policy Guidance21 Apr
  5. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  7. 2025Process Reinforcement through Implicit Rewards3 Feb
  8. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  9. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations14 Dec
  10. 2023Let's Verify Step by Step31 May
  11. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  12. 2022Constitutional AI: Harmlessness from AI Feedback15 Dec
1 more
  1. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 16

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  5. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  8. 2024Self-Rewarding Language Modelscited by 654
  9. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  10. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  11. 2024TravelPlanner: A Benchmark for Real-World Planning with Language Agentscited by 385
  12. 2021MiniF2F: a cross-system benchmark for formal Olympiad-level mathematicscited by 380
4 more
  1. 2022Self-critiquing models for assisting human evaluatorscited by 372
  2. 2022Generating Sequences by Learning to Self-Correctcited by 271
  3. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 162
  4. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Rollout-Level Advantage-Prioritized Experience Replay for GRPO
  2. 2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
  3. 2026Learn from the Gap: Differential-Aware Advantage Pruning with Adaptive Rollout Sampling for GRPO
  4. 2026Gradient Extrapolation-Based Policy Optimization
  5. 2025TIC-GRPO: Provable and Efficient Optimization for Reinforcement Learning from Human Feedback
  6. 2026How Off-Policy Can GRPO Be? Mu-GRPO for Efficient LLM Reinforcement Learning
  7. 2026Prism-GRPO: Faster VLA Policy Optimization via Splitting Same-outcome Groups
  8. 2025Slow-Fast Policy Optimization: Reposition-Before-Update for LLM Reasoning