mlsanityData to 7 Oct 2026

FlowBalance: Verifier-Grounded Self-Improvement from On-Policy Reasoning Experience

arXiv 2609.03241 ↗First posted 3 Sep 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

A reasoning model can improve from its own on-policy experience, but this inner loop is fragile: terminal verifiers provide reliable yet sparse supervision, while dense same-model guidance can reinforce false confidence or overconcentrate learning on a narrow solution mode. We introduce FlowBalance, a verifier-grounded self-improvement method that learns a normalized distribution over complete responses. For each on-policy trajectory, a frozen training-time view of the same policy uses privileged context to produce token-level log-probability gains, which are aggregated into a trajectory-level self-guidance score. FlowBalance calibrates this score with the verifier-derived group advantage: guidance is retained on positive-advantage trajectories, reversed on negative-advantage trajectories, and disabled when the rollout group provides no outcome preference. The resulting energy exponentially reweights a reference policy, and profiled trajectory balance fits the normalized target with one log-partition estimate per rollout group. This realizes outcome-calibrated self-guidance via trajectory balance, without a separate token-level imitation loss. Our analysis establishes within-group contrast preservation, a minimum-change reverse-KL characterization, monotonic verifier control of target reward, and an exact correction against false-positive self-guidance on rejected responses. On mathematical reasoning, FlowBalance improves average performance over FlowRL on both Qwen3-4B and Qwen3-8B, while also improving training speed and stability, avoiding direct OPSD's response-length collapse, and exhibiting higher correct-strategy diversity in a controlled AIME24 diagnostic.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 30 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

21 of 30 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026ROSD: Reflective On-Policy Self-Distillation for Language Model Reasoning across Domains27 May
  2. 2026Tailoring Teaching to Aptitude: Direction-Adaptive Self-Distillation for LLM Reasoning21 May
  3. 2026Rebellious Student: Reversing Teacher Signals for Reasoning Exploration with Self-Distilled RLVR11 May
  4. 2026OPSD Compresses What RLVR Teaches: A Post-RL Compaction Stage for Reasoning Models7 May
  5. 2026Why Does Self-Distillation (Sometimes) Degrade the Reasoning Capability of LLMs?25 Mar

Other references · 16

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  3. 2021Flow Network based Generative Models for Non-Iterative Diverse Candidate Generationcited by 384
  4. 2021GFlowNet Foundationscited by 286
  5. 2022Trajectory balance: Improved credit assignment in GFlowNetscited by 244
  6. 2022Robust Reinforcement Learning using Offline Datacited by 95
  7. 2024Learning Dynamics of LLM Finetuningcited by 93
  8. 2025Evolving Language Models without Labels: Majority Drives Selection, Novelty Promotes Variationcited by 44
  9. 2024Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examplescited by 29
  10. 2026Asymmetric On-Policy Distillation: Bridging Exploitation and Imitation at the Token Levelcited by 29
  11. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimizationcited by 17
  12. 2026Group Distributionally Robust Optimization-Driven Reinforcement Learning for LLM Reasoningcited by 9
4 more
  1. 2024Neural Algorithmic Reasoning with Multiple Correct Solutionscited by 4
  2. 2025Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Taskscited by 2
  3. 2026Reasoning or Memorization? Direction-Aware Diversity Exploration in LLM Reinforcement Learningcited by 2
  4. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewardscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026R2 Flow: Recursive Self-Improvement via Recursive Skill Evolution
  2. 2026VeriGate: Verifier-Gated Step-Level Supervision for GRPO
  3. 2026ReNIO: Reweighting Negative Trajectory Importance for LLM On-Policy Distillation
  4. 2026Verifier-Induced Support Reshaping in On-Policy Optimization
  5. 2026Unmasking On-Policy Distillation: Where It Helps, Where It Hurts, and Why
  6. 2025Bootstrapping Task Spaces for Self-Improvement
  7. 2026PHF: Privileged Hidden Flow for On-Policy Self-Distillation
  8. 2026SIPO: Unifying Reinforcement Learning with On-Policy Self-Distillation