mlsanityData to 14 Oct 2026

TRE: Encouraging Exploration in the Trust Region

arXiv 2602.03635 ↗First posted 3 Feb 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Entropy regularization is a standard technique in reinforcement learning (RL) to enhance exploration, yet it yields negligible effects or even degrades performance in Large Language Models (LLMs). We attribute this failure to the cumulative tail risk inherent to LLMs with massive vocabularies and long generation horizons. In such environments, standard global entropy maximization indiscriminately dilutes probability mass into the vast tail of invalid tokens rather than focusing on plausible candidates, thereby disrupting coherent reasoning. To address this, we propose Trust Region Entropy (TRE), a method that encourages exploration strictly within the model's trust region. Extensive experiments across mathematical reasoning (MATH), combinatorial search (Countdown), and preference alignment (HH) tasks demonstrate that TRE consistently outperforms vanilla PPO, standard entropy regularization, and other exploration baselines. Our code is available at https://github.com/WhyChaos/TRE-Encouraging-Exploration-in-the-Trust-Region.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 14 of its 15 references that are in the map. First posted on arXiv 3 Feb 2026.

Builds on

14 of 15 references.

References · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  8. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  9. 2015Trust Region Policy Optimizationcited by 2,580
  10. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  11. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 559
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
2 more
  1. 2018Understanding the impact of entropy on policy optimizationcited by 176
  2. 2025State Entropy Regularization for Robust Reinforcement Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Entropy Regularization: Adaptive Coefficient Unlocks Its Potential for LLM Reinforcement Learning
  2. 2025Reasoning with Exploration: An Entropy Perspective
  3. 2025TROLL: Trust Regions improve Reinforcement Learning for Large Language Models
  4. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  5. 2026Asymmetric Advantage Modulation Calibrates Entropy Dynamics in RLVR
  6. 2026Beyond Uniform Token-Level Trust Region in LLM Reinforcement Learning
  7. 2025Exploration vs Exploitation: Rethinking RLVR through Clipping, Entropy, and Spurious Reward
  8. 2025Beyond High-Entropy Exploration: Correctness-Aware Low-Entropy Segment-Based Advantage Shaping for Reasoning LLMs