mlsanityData to 14 Oct 2026

Explore with Memory, Learn without It: In-Context Exploration and Self-Distillation for LLM Reasoning

arXiv 2610.17734 ↗First posted 13 Oct 2026cs.LG cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Exploration bounds learning efficiency in reinforcement learning with verifiable rewards (RLVR). Current methods explore mostly through independent sampling, without fully using the information gathered along the way, such as known wrong answers and environment feedback. We propose LEAF, which injects a dynamic, problem-specific memory into the context before every rollout. The memory contains the most recent response, its reward, and other environment feedback. It is updated after every verifier call. We decouple policy updates from rollouts and update the policy through self-distillation. On four benchmarks and two open models, LEAF accelerates exploration by about 4x compared with a random sampling baseline. On unseen test sets, it improves accuracy over the strongest baseline by 1.59 and 1.83 percentage points on the two models, while using less compute than any RL baseline. These results highlight the potential of LEAF in improving both exploration and learning efficiency. Code is available at: https://github.com/on1262/LEAF

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 35 of its 48 references that are in the map. First posted on arXiv 13 Oct 2026.

Builds on

35 of 48 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Provable and Practical In-Context Policy Optimization for Self-Improvement2 Mar
  2. 2026Reasoning Cache: Continual Improvement Over Long Horizons via Short-Horizon RL3 Feb
  3. 2025MURPHY: Feedback-Aware GRPO with Retrospective Credit Assignment for Multi-Turn Code Generation11 Nov
  4. 2025Beyond Markovian: Reflective Exploration via Bayes-Adaptive RL for LLM Reasoning26 May
  5. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  6. 2023Let's Verify Step by Step31 May
  7. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 28

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  7. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  8. 2023On-Policy Distillation of Language Models: Learning from Self-Generated Mistakescited by 939
  9. 2025Group Sequence Policy Optimizationcited by 835
  10. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  11. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  12. 2026Reinforcement Learning via Self-Distillationcited by 332
16 more
  1. 2024VinePPO: Refining Credit Assignment in RL Training of LLMscited by 183
  2. 2026Skill-SD: Skill-Conditioned Self-Distillation for Multi-turn LLM Agentscited by 90
  3. 2026Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervisioncited by 68
  4. 2025e3: Learning to Explore Enables Extrapolation of Test-Time Compute for LLMscited by 59
  5. 2026Expanding the Capabilities of Reinforcement Learning via Text Feedbackcited by 53
  6. 2024SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correctioncited by 40
  7. 2026SEED: Self-Evolving On-Policy Distillation for Agentic Reinforcement Learningcited by 32
  8. 2025Reward Is Enough: LLMs Are In-Context Reinforcement Learnerscited by 31
  9. 2025Multi-Turn Code Generation Through Single-Step Rewardscited by 28
  10. 2026Exploratory Memory-Augmented LLM Agent via Hybrid On- and Off-Policy Optimizationcited by 19
  11. 2024Think Thrice Before You Act: Progressive Thought Refinement in Large Language Modelscited by 7
  12. 2026Procedural Memory Distillation: Online Reflection for Self-Improving Language Modelscited by 6
  13. 2026Distill Skills into Weights, Not Prompts: Abstract Skills as Privileged Signals for On-Policy Self-Distillationcited by 5
  14. 2026MulFeRL: Enhancing Reinforcement Learning with Verbal Feedback in a Multi-turn Loopcited by 4
  15. 2026Instruction-Conditioned Exploration for Reinforcement Learning with Self-Distillation to an Unconditioned Policycited by 2
  16. 2026DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimizationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Decoupling Exploration from Optimization in RLVR
  2. 2025Low-probability Tokens Sustain Exploration in Reinforcement Learning with Verifiable Reward
  3. 2025From Trial-and-Error to Improvement: A Systematic Analysis of LLM Exploration Mechanisms in RLVR
  4. 2025Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning
  5. 2026Offline Exploration-Aware Fine-Tuning for Long-Chain Mathematical Reasoning
  6. 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
  7. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  8. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuning