mlsanityData to 7 Oct 2026

D2Evo: Dual Difficulty-Aware Self-Evolution for Data-Efficient Reinforcement Learning

arXiv 2605.17037 ↗First posted 16 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reinforcement learning (RL) has demonstrated potential for enhancing reasoning in large language models (LLMs). However, effective RL training, which requires medium-difficulty training samples, faces two fundamental challenges: Effective Data Scarcity and Dynamic Difficulty Shifts, where medium-difficulty samples are scarce and become trivial as models improve. Existing methods mitigate this scarcity to some extent by generating training samples. However, these approaches suffer from anchor-free generation, ignoring co-evolution, and difficulty mismatch. To address these issues, we propose D2Evo, a Dual Difficulty-aware self-Evolution RL framework. In each iteration, our method mines medium-difficulty anchors based on the current Solver's capability, trains the Questioner to generate diverse questions at appropriate difficulty levels, and jointly optimizes both components to enable progressive reasoning gains. Extensive experiments demonstrate that D2Evo outperforms existing methods on mathematical reasoning benchmarks with fewer than 2K real mathematical samples, and exhibits strong generalization on general reasoning benchmarks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 44 of its 50 references that are in the map. First posted on arXiv 16 May 2026.

Builds on

44 of 50 references.

Closest ancestors · 17

Same direction, newest first

  1. 2026Harder Is Better: Boosting Mathematical Reasoning via Difficulty-Aware GRPO and Multi-Aspect Question Reformulation28 Jan
  2. 2025AdaCuRL: Adaptive Curriculum Reinforcement Learning with Invalid Sample Mitigation and Historical Revisiting12 Nov
  3. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentation17 Jul
  4. 2025No Free Lunch: Rethinking Internal Feedback for LLM Reasoning20 Jun
  5. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoning10 Jun
  6. 2025Maximizing Confidence Alone Improves Reasoning28 May
  7. 2025The Unreasonable Effectiveness of Entropy Minimization in LLM Reasoning21 May
  8. 2025Learning Like Humans: Advancing LLM Reasoning Capabilities via Adaptive Difficulty Curriculum Learning and Expert-Guided Self-Reformulation13 May
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Data6 May
  10. 2025Efficient Reinforcement Finetuning via Adaptive Curriculum Learning7 Apr
  11. 2025Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning4 Apr
  12. 2025GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning3 Apr
5 more
  1. 2025Reinforcement Learning for Reasoning in Small LLMs: What Works and What Doesn't20 Mar
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization4 Jan
  4. 2023Let's Verify Step by Step31 May
  5. 2017Proximal Policy Optimization Algorithms20 Jul

Other references · 27

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  10. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  11. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  12. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,099
15 more
  1. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
  2. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thoughtcited by 334
  3. 2022Large Language Models Are Reasoning Teacherscited by 318
  4. 2025R-Zero: Self-Evolving Reasoning LLM from Zero Datacited by 210
  5. 2025BIG-Bench Extra Hardcited by 123
  6. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 78
  7. 2025Boosting the Generalization and Reasoning of Vision Language Models with Curriculum Reinforcement Learningcited by 71
  8. 2025Language Self-Play For Data-Free Trainingcited by 42
  9. 2025Long-Short Chain-of-Thought Mixture Supervised Fine-Tuning Eliciting Efficient Reasoning in Large Language Modelscited by 37
  10. 2025MathFusion: Enhancing Mathematical Problem-solving of LLM through Instruction Fusioncited by 30
  11. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 27
  12. 2025Socratic-Zero : Bootstrapping Reasoning via Data-Free Agent Co-evolutioncited by 24
  13. 2025HS-STaR: Hierarchical Sampling for Self-Taught Reasoners via Difficulty Estimation and Budget Reallocationcited by 18
  14. 2026CoEvolve: Training LLM Agents via Agent-Data Mutual Evolutioncited by 14
  15. 2025Infi-MMR: Curriculum-based Unlocking Multimodal Reasoning via Phased Reinforcement Learning in Multimodal Small Language Modelscited by 10

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback
  2. 2026DARE: Difficulty-Adaptive Reinforcement Learning with Co-Evolved Difficulty Estimation
  3. 2026STRETCH the Boundaries: A Unified Self-Taught Framework for Progressive LLM Evolution
  4. 2025How Difficulty-Aware Staged Reinforcement Learning Enhances LLMs' Reasoning Capabilities: A Preliminary Experimental Study
  5. 2026Learning to Self-Evolve
  6. 2026Direct Self-Evolving Optimization: Evolving LLMs without Challenger Training
  7. 2026SAGE: Multi-Agent Self-Evolution for LLM Reasoning
  8. 2025EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning