mlsanityData to 7 Oct 2026

QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning

arXiv 2602.04620 ↗First posted 4 Feb 2026cs.LGcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

GRPO-style reinforcement learning (RL)-based LLM fine-tuning algorithms have recently gained popularity. Relying on heuristic trust-region approximations, however, they can lead to brittle optimization behavior, as global importance-ratio clipping and group-wise normalization fail to regulate samples whose importance ratios fall outside the clipping range. We propose Query-Adaptive Trust-Region policy Optimization (QUATRO), which directly enforces trust-region constraints through a principled optimization. This yields a clear and interpretable objective that enables explicit control over policy updates and stable, entropy-controlled optimization, with a stabilizer terms arising intrinsically from the exact trust-region formulation. Empirically verified on diverse mathematical reasoning benchmarks, QUATRO shows stable training under increased policy staleness and aggressive learning rates, maintaining well-controlled entropy throughout training.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 50 of its 60 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

50 of 60 references.

Closest ancestors · 26

Same direction, newest first

  1. 2025Scaf-GRPO: Scaffolded Group Relative Policy Optimization for Enhancing LLM Reasoning22 Oct
  2. 2025XRPO: Pushing the limits of GRPO with Targeted Exploration and Exploitation8 Oct
  3. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning4 Oct
  4. 2025LSPO: Length-aware Dynamic Sampling for Policy Optimization in LLM Reasoning1 Oct
  5. 2025Prosperity before Collapse: How Far Can Off-Policy RL Reach with Stale Data on LLMs?1 Oct
  6. 2025RiskPO: Risk-based Policy Optimization via Verifiable Reward for LLM Post-Training1 Oct
  7. 2025Quantile Advantage Estimation: Stabilizing RLVR for LLM Reasoning26 Sep
  8. 2025CE-GPPO: Coordinating Entropy via Gradient-Preserving Clipping Policy Optimization in Reinforcement Learning25 Sep
  9. 2025DCPO: Dynamic Clipping Policy Optimization2 Sep
  10. 2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimization11 Aug
  11. 2025Geometric-Mean Policy Optimization28 Jul
  12. 2025Reasoning with Exploration: An Entropy Perspective17 Jun
14 more
  1. 2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selection5 Jun
  2. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening3 Jun
  3. 2025AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning30 May
  4. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models28 May
  5. 2025Revisiting Group Relative Policy Optimization: Insights into On-Policy and Off-Policy Training28 May
  6. 2025On the Design of KL-Regularized Policy Gradient Algorithms for LLM Reasoning23 May
  7. 2025SEED-GRPO: Semantic Entropy Enhanced GRPO for Uncertainty-Aware Policy Optimization18 May
  8. 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization8 Apr
  9. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  11. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  12. 2023Let's Verify Step by Step31 May
  13. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  14. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 24

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2021Convex optimizationcited by 2,479
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  12. 2020Mirror Descent Policy Optimizationcited by 102
12 more
  1. 2024OlympicArena: Benchmarking Multi-discipline Cognitive Reasoning for Superintelligent AIcited by 70
  2. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  3. 2025BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via Balanced Policy Optimization with Adaptive Clippingcited by 47
  4. 2025Harnessing Uncertainty: Entropy-Modulated Policy Gradients for Long-Horizon LLM Agentscited by 43
  5. 2023Generative Modeling through the Semi-dual Formulation of Unbalanced Optimal Transportcited by 39
  6. 2025Asymmetric REINFORCE for off-Policy Reinforcement Learning: Balancing positive and negative rewardscited by 29
  7. 2021Differentiable Trust Region Layers for Deep Reinforcement Learningcited by 24
  8. 2024Scalable Wasserstein Gradient Flow for Generative Modeling through Unbalanced Optimal Transportcited by 22
  9. 2025Trust Region Constrained Measure Transport in Path Space for Stochastic Optimal Control and Inferencecited by 22
  10. 2025Proximal Diffusion Neural Samplercited by 16
  11. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMscited by 4
  12. 2025Entropy Regularizing Activation: Boosting Continuous Control, Large Language Models, and Image Classification with Activation as Entropy Constraintscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Scaling Reinforcement Learning for Diffusion Models via Velocity Matchingcited by 1
  2. 2026When KL Regularization Misfires in Group Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Trust-Region Adaptive Policy Optimization
  2. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
  3. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learning
  4. 2026Ratio-Variance Regularized Policy Optimization for Efficient LLM Fine-tuning
  5. 2026BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
  6. 2026Ratio-Variance Regularized Policy Optimization
  7. 2023Supported Trust Region Optimization for Offline Reinforcement Learning
  8. 2026Trust Region Q Adjoint Matching