# Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policies

arXiv: https://arxiv.org/abs/2512.19673 · HTML: https://mlsanity.com/paper/2512.19673/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Dec 2025.

## Built on by (most cited first, 23 of 23)

- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [ResAdapt: Adaptive Resolution for Efficient Multimodal Reasoning](https://mlsanity.com/paper/2603.28610/) (2026)
- [Unveiling Implicit Advantage Symmetry: Why GRPO Struggles with Exploration and Difficulty Adaptation](https://mlsanity.com/paper/2602.05548/) (2026)
- [Towards a Mechanistic Understanding of Large Reasoning Models: A Survey of Training, Inference, and Failures](https://mlsanity.com/paper/2601.19928/) (2026)
- [When Models Learn to Ask Why: Adaptive Causal Reasoning for Trustworthy Medical Vision-Language Models](https://mlsanity.com/paper/2603.23085/) (2026)
- [DyJR: Preserving Diversity in Reinforcement Learning with Verifiable Rewards via Dynamic Jensen-Shannon Replay](https://mlsanity.com/paper/2603.16157/) (2026)
- [SocraticPO: Policy Optimization via Interactive Guidance](https://mlsanity.com/paper/2606.09887/) (2026)
- [HogVul: Black-box Adversarial Code Generation Framework Against LM-based Vulnerability Detectors](https://mlsanity.com/paper/2601.05587/) (2026)
- [SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation](https://mlsanity.com/paper/2603.16161/) (2026)
- [From P(y|x) to P(y): Investigating Reinforcement Learning in Pre-train Space](https://mlsanity.com/paper/2604.14142/) (2026)
- [Uncertainty Propagation in LLM-Based Systems](https://mlsanity.com/paper/2604.23505/) (2026)
- [Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learning](https://mlsanity.com/paper/2602.13562/) (2026)
- [Reasoning-preserved Efficient Distillation of Large Language Models via Activation-aware Initialization](https://mlsanity.com/paper/2605.29327/) (2026)
- [PAEC: Position-Aware Entropy Calibration for LLM Reasoning in RLVR](https://mlsanity.com/paper/2606.08543/) (2026)
- [Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning](https://mlsanity.com/paper/2606.21943/) (2026)
- [SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning for LLMs](https://mlsanity.com/paper/2608.03573/) (2026)
- [The Past Is Not Past: Memory-Enhanced Dynamic Reward Shaping](https://mlsanity.com/paper/2604.11297/) (2026)
- [OISD: On-Policy Internal Self-Distillation of Language Models](https://mlsanity.com/paper/2605.29089/) (2026)
- [Depth-Entropy Guided Sampling for Training-Free LLM Reasoning](https://mlsanity.com/paper/2607.09693/) (2026)
- [Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning](https://mlsanity.com/paper/2602.11748/) (2026)
- [Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning](https://mlsanity.com/paper/2604.15830/) (2026)
- [Breaking the Impasse: Dual-Scale Evolutionary Policy Training for Social Language Agents](https://mlsanity.com/paper/2605.08721/) (2026)
- [SAEScientist-Bench: Can AI Agents Conduct Autonomous SAE Interpretability Research?](https://mlsanity.com/paper/2609.09113/) (2026)

## Abstract (arXiv)

Existing reinforcement learning (RL) approaches treat large language models (LLMs) as a unified policy, overlooking their internal mechanisms. In this paper, we decompose the LLM-based policy into Internal Layer Policies and Internal Modular Policies via the Transformer's residual stream. Our entropy analysis of internal policy reveals distinct patterns: (1) universally, internal policies evolve from high-entropy exploration in early layers to deterministic refinement in the top layers; and (2) Qwen exhibits an explicit progressive reasoning structure, contrasting with the abrupt convergence in Llama. Furthermore, we discover that optimizing internal layers induces feature refinement, forcing lower layers to capture high-level reasoning representations early. Motivated by these findings, we propose Bottom-up Policy Optimization (BuPO), a novel RL paradigm that reconstructs the LLM's reasoning foundation from the bottom up by optimizing internal layers in early stages. Extensive experiments on complex reasoning benchmarks demonstrate the effectiveness of BuPO.