mlsanityData to 7 Oct 2026

Trust-Region Adaptive Policy Optimization

arXiv 2512.17636 ↗First posted 19 Dec 2025cs.LG cs.AIcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Post-training methods, especially Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL), play an important role in improving large language models' (LLMs) complex reasoning abilities. However, the dominant two-stage pipeline (SFT then RL) suffers from a key inconsistency: SFT enforces rigid imitation that suppresses exploration and induces forgetting, limiting RL's potential for improvements. We address this inefficiency with TRAPO (Trust-Region Adaptive Policy Optimization), a hybrid framework that interleaves SFT and RL within each training instance by optimizing SFT loss on expert prefixes and RL loss on the model's own completions, unifying external supervision and self-exploration. To stabilize training, we introduce Trust-Region SFT (TrSFT), which minimizes forward KL divergence inside a trust region but attenuates optimization outside, effectively shifting toward reverse KL and yielding stable, mode-seeking updates favorable for RL. An adaptive prefix-selection mechanism further allocates expert guidance based on measured utility. Experiments on five mathematical reasoning benchmarks show that TRAPO consistently surpasses standard SFT, RL, and SFT-then-RL pipelines, as well as recent state-of-the-art approaches, establishing a strong new paradigm for reasoning-enhanced LLMs.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 48 references that are in the map. First posted on arXiv 19 Dec 2025.

Builds on

41 of 48 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025AMFT: Aligning LLM Reasoners by Meta-Learning the Optimal Imitation-Exploration Balance9 Aug
  2. 2025Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling2 Jul
  3. 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
  4. 2025Learning What Reinforcement Learning Can't: Interleaved Online Fine-Tuning for Hardest Questions9 Jun
  5. 2025UFT: Unifying Supervised and Reinforcement Fine-Tuning22 May
  6. 2025Learning to Reason under Off-Policy Guidance21 Apr
  7. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
  8. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  9. 2025VAPO: Efficient and Reliable Reinforcement Learning for Advanced Reasoning Tasks7 Apr
  10. 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
  11. 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
  12. 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
8 more
  1. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  2. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRs3 Mar
  3. 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
  4. 2025Process Reinforcement through Implicit Rewards3 Feb
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  6. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMs22 Jan
  7. 2024Training Large Language Models for Reasoning through Reverse Curriculum Reinforcement Learning8 Feb
  8. 2023Let's Verify Step by Step31 May

Other references · 21

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,143
9 more
  1. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
  2. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,057
  3. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 840
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  5. 2018Behavioral Cloning from Observationcited by 482
  6. 2025SFT or RL? An Early Investigation into Training R1-Like Reasoning Large Vision-Language Modelscited by 209
  7. 2019Reverse KL-Divergence Training of Prior Networks: Improved Uncertainty and Adversarial Robustnesscited by 104
  8. 2025Towards a Unified View of Large Language Model Post-Trainingcited by 36
  9. 2021KKT Conditions, First-Order and Second-Order Optimization, and Distributed Optimization: Tutorial and Surveycited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failurescited by 5
  2. 2026When Metrics Reward the Worst Translations: Internalizing Cultural Reasoning for Social Media Translation Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Bridging SFT and RL: Dynamic Policy Optimization for Robust Reasoning
  2. 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
  3. 2026CARE-RFT: Confidence-Anchored Reinforcement Finetuning for Reliable Reasoning in Large Language Models
  4. 2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-training
  5. 2025Beyond Two-Stage Training: Cooperative SFT and RL for LLM Reasoning
  6. 2025Trust Region Masking for Long-Horizon LLM Reinforcement Learning
  7. 2026Rethinking the Divergence Regularization in LLM RL
  8. 2026On-Policy Supervised Fine-Tuning for Efficient Reasoning