mlsanityData to 7 Oct 2026

Think Longer to Explore Deeper: Learn to Explore In-Context via Length-Incentivized Reinforcement Learning

arXiv 2602.11748 ↗First posted 12 Feb 2026cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Achieving effective test-time scaling requires models to engage in In-Context Exploration -- the intrinsic ability to generate, verify, and refine multiple reasoning hypotheses within a single continuous context. Grounded in State Coverage theory, our analysis identifies a critical bottleneck to enabling this capability: while broader state coverage requires longer reasoning trajectories, the probability of sampling such sequences decays exponentially during autoregressive generation, a phenomenon we term the ``Shallow Exploration Trap''. To bridge this gap, we propose Length-Incentivized Exploration(method). This simple yet effective recipe explicitly encourages models to explore more via a length-based reward coupled with a redundancy penalty, thereby maximizing state coverage in two-step manner. Comprehensive experiments across different models (Qwen3, Llama) demonstrate that method effectively incentivize in-context exploration. As a result, our method achieves an average improvement of 4.4% on in-domain tasks and a 2.7% gain on out-of-domain benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 45 of its 59 references that are in the map. First posted on arXiv 12 Feb 2026.

Builds on

45 of 59 references.

Closest ancestors · 20

Same direction, newest first

  1. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens24 Aug
  2. 2025gpt-oss-120b & gpt-oss-20b Model Card8 Aug
  3. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMs10 Jun
  4. 2025OThink-R1: Intrinsic Fast/Slow Thinking Mode Switching for Over-Reasoning Mitigation3 Jun
  5. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Time30 May
  6. 2025AdaCtrl: Towards Adaptive and Controllable Reasoning via Difficulty-Aware Budgeting24 May
  7. 2025Learn to Reason Efficiently with Adaptive Length-based Reward Shaping21 May
  8. 2025AdaptThink: Reasoning Models Can Learn When to Think19 May
  9. 2025Scalable Chain of Thoughts via Elastic Reasoning8 May
  10. 2025L1: Controlling How Long A Reasoning Model Thinks With Reinforcement Learning6 Mar
  11. 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning25 Feb
  12. 2025CoT-Valve: Length-Compressible Chain-of-Thought Tuning13 Feb
8 more
  1. 2025Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling10 Feb
  2. 2025Demystifying Long Chain-of-Thought Reasoning in LLMs5 Feb
  3. 2025s1: Simple test-time scaling31 Jan
  4. 2025Think Smarter not Harder: Adaptive Reasoning with Inference Aware Optimization29 Jan
  5. 2024C3oT: Generating Shorter Chain-of-Thought without Compromising Effectiveness16 Dec
  6. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  8. 2024Stream of Search (SoS): Learning to Search in Language1 Apr

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2023Let's Verify Step by Stepcited by 3,598
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
13 more
  1. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  2. 2025Group Sequence Policy Optimizationcited by 788
  3. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  4. 2025Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoningcited by 540
  5. 2025Cognitive Behaviors that Enable Self-Improving Reasoners, or, Four Habits of Highly Effective STaRscited by 373
  6. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  7. 2025Reasoning with Exploration: An Entropy Perspectivecited by 277
  8. 2025DeepMath-103K: A Large-Scale, Challenging, Decontaminated, and Verifiable Mathematical Dataset for Advancing Reasoningcited by 261
  9. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 158
  10. 2025OctoThinker: Mid-training Incentivizes Reinforcement Learning Scalingcited by 110
  11. 2025RLPR: Extrapolating RLVR to General Domains without Verifierscited by 83
  12. 2025What Characterizes Effective Reasoning? Revisiting Length, Review, and Structure of CoTcited by 24
  13. 2025Bottom-up Policy Optimization: Your Language Model Policy Secretly Contains Internal Policiescited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Extending Test-Time Scaling: A 3D Perspective with Context, Batch, and Turn
  2. 2025ReLaX: Reasoning with Latent Exploration for Large Reasoning Models
  3. 2026Learning to Explore: Scaling Agentic Reasoning via Exploration-Aware Policy Optimization
  4. 2026Mitigating the Length-Scaling Tax with Online Distillation
  5. 2024Can large language models explore in-context?
  6. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Training
  7. 2026When Does In-Context Search Help? A Sampling-Complexity Theory of Reflection-Driven Reasoning
  8. 2025Disentangling Exploration of Large Language Models by Optimal Exploitation