mlsanityData to 14 Oct 2026

FSPO: Policy-Consistent Risk and Pareto-Feasible Control for Budgeted LLM RL Post-Training

arXiv 2610.02828 ↗First posted 2 Oct 2026cs.AI cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Adaptive LLM reinforcement-learning post-training changes multiple training actuators online, including rollout temperature, group size, clipping, KL regularization, verifier allocation, and update budget. Three coupled issues remain unresolved. A future-risk model trained from behavior trajectories need not estimate the risk induced by the controller that will be deployed; a score calibrated on logged state-action pairs can become miscalibrated after selective action choice; and independent per-resource minimum costs do not in general certify a feasible multi-resource continuation. We introduce FSPO, a feedback-state controller for budgeted LLM RL post-training that addresses these issues jointly. FSPO learns a policy-consistent risk-to-go model whose Bellman target follows the same frozen controller used for future decisions, together with a long-horizon utility model. Decision-conditioned trajectory calibration (DCTC) calibrates risk on cross-fitted trajectories generated by actions selected by provisional controllers. A Pareto resource continuation certificate (PRCC) admits an action only when a non-dominated cumulative reservation remains feasible over the residual horizon. Under a matched GRPO resource envelope, FSPO reaches 66.11% held-out and 59.43% OOD accuracy, compared with 64.47% and 57.03% for PB2, the strongest evaluated adaptive baseline. Three paired training seeds give gains of +2.42 and +3.19 percentage points over the contextual bandit on held-out and OOD evaluation. Under high behavior-deployment mismatch, policy-consistent risk lowers selected-decision ECE from 0.108 to 0.053; DCTC lowers it from 0.039 to 0.022 at matched acceptance; PRCC removes false-feasible admissions on an 18-action catalog (0.197rightarrow0.000); and enabling all three components reduces trajectory failure from 0.181 to 0.083 in a factorial ablation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 37 of its 51 references that are in the map. First posted on arXiv 2 Oct 2026.

Builds on

37 of 51 references.

References · 37

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2023Let's Verify Step by Stepcited by 3,789
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  6. 2015Trust Region Policy Optimizationcited by 2,580
  7. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  8. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  9. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  10. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  11. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  12. 2015Prioritized Experience Replaycited by 1,099
25 more
  1. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  2. 2017Constrained Policy Optimizationcited by 983
  3. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  4. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  5. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  6. 2017Deep Reinforcement Learning that Matterscited by 754
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  8. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  9. 2017Population Based Training of Neural Networkscited by 367
  10. 2019SelectiveNet: A Deep Neural Network with an Integrated Reject Optioncited by 350
  11. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  12. 2019Batch Policy Learning under Constraintscited by 244
  13. 2014A Survey of Multi-Objective Sequential Decision-Makingcited by 242
  14. 2022Defining and Characterizing Reward Hackingcited by 203
  15. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  16. 2018Meta-Gradient Reinforcement Learningcited by 159
  17. 2017Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximationcited by 121
  18. 2020Learning to be Safe: Deep RL with a Safety Criticcited by 116
  19. 2020Conservative Safety Critics for Explorationcited by 113
  20. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 101
  21. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  22. 2020Provably Efficient Online Hyperparameter Optimization with Population-Based Banditscited by 63
  23. 2026Using Common Random Numbers for Simulation-based Planning with Rolloutscited by 5
  24. 2026Where Should RL Post-Training Compute Go? Model Size, Search, Learning, and Feedbackcited by 2
  25. 2026Cutting AI Datacenter Energy with Reinforcement Learning: Measured Power Control of LLM Training from One GPU to the Fleetcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Cross-Epoch Adaptive Rollout Optimization for RL Post-Training
  2. 2026Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
  3. 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
  4. 2026RL Excursions during Pre-Training: Re-examining Policy Optimization for LLM training
  5. 2026EVPO: Explained Variance Policy Optimization for Adaptive Critic Utilization in LLM Post-Training
  6. 2026Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training
  7. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning
  8. 2025LLM-Guided Reinforcement Learning: Addressing Training Bottlenecks through Policy Modulation