mlsanityData to 14 Oct 2026

Breaking the Solver Bottleneck: Training Task Generators at the Learnable Frontier

arXiv 2606.18284 ↗First posted 10 Jun 2026cs.LG cs.AI cs.CLcited by 3

On the map: Deep reinforcement learning methods → RL training of language and agent models

Abstract (arXiv)

The limiting resource for training agents via reinforcement learning (RL) is increasingly frontier task supply: valid, solvable tasks just difficult enough to train the current model. As reasoning and agentic models improve, fixed task distributions saturate, while naive synthetic generation yields tasks that are trivial, impossible, or ill-posed. Training a task generator with RL to optimize validity and learnability can address this bottleneck, but direct optimization requires repeated solver rollouts per candidate. For software-engineering (SWE) tasks, a single rollout can take tens of minutes; solver-in-the-loop generator training is intractable. We introduce PROPEL, a solver-amortized framework for training task generators at the targeted solve rate. PROPEL trains a lightweight activation probe on a one-time labeled corpus of generated tasks and solver outcomes. The probe predicts target-solver pass rate from a frozen generator reference model and serves as a proxy for solve rate during generator optimization, reducing generator evaluation to a single forward pass. Across math, code, and software-engineering at multiple model scales, PROPEL shifts generation toward the targeted solve rate: for coding, tasks generated at the learnable frontier increase from 10.1% rightarrow 20.0% for a Qwen2.5-3B-Instruct solver and from 5.3% rightarrow 12.6% for a Qwen2.5-7B-Instruct solver. For SWE, PROPEL increases the share of generations at the targeted solve rate from 9.8% rightarrow 19.6% for Qwen3.5-27B on repositories not seen during training of probe and generator.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods. Builds on 43 of its 52 references that are in the map. First posted on arXiv 10 Jun 2026.

Builds on

43 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026Features as Rewards: Scalable Supervision for Open-Ended Tasks via Interpretability10 Feb
  2. 2026Silence the Judge: Reinforcement Learning with Self-Verifier via Latent Geometric Clustering13 Jan
  3. 2025Reinforcement Learning from Human Feedback16 Apr

Other references · 40

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  10. 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
  11. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
  12. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
28 more
  1. 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Modelscited by 194
  2. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
  3. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  4. 2025Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversitycited by 105
  5. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  6. 2024Open-Endedness is Essential for Artificial Superhuman Intelligencecited by 80
  7. 2019Jointly Measuring Diversity and Quality in Text Generation Modelscited by 61
  8. 2025QuestA: Expanding Reasoning Capacity in LLMs via Question Augmentationcited by 44
  9. 2025SwS: Self-aware Weakness-driven Problem Synthesis in Reinforcement Learning for LLM Reasoningcited by 42
  10. 2025Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewardscited by 42
  11. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 40
  12. 2025No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probescited by 38
  13. 2025KL-Regularized Reinforcement Learning is Designed to Mode Collapsecited by 31
  14. 2025BugPilot: Complex Bug Generation for Efficient Learning of SWE Skillscited by 30
  15. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 23
  16. 2025RLAD: Training LLMs to Discover Abstractions for Solving Reasoning Problemscited by 21
  17. 2025SWE-Flow: Synthesizing Software Engineering Data in a Test-Driven Mannercited by 20
  18. 2025Training Versatile Coding Agents in Synthetic Environmentscited by 13
  19. 2025Reward Model Overoptimisation in Iterated RLHFcited by 12
  20. 2025DISCOVER: Automated Curricula for Sparse-Reward Reinforcement Learningcited by 12
  21. 2025CLUE: Non-parametric Verification from Experience via Hidden-State Clusteringcited by 12
  22. 2025Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Samplingcited by 9
  23. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 9
  24. 2025SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoningcited by 8
  25. 2025Temporal Predictors of Outcome in Reasoning Language Modelscited by 6
  26. 2026Hybrid-Gym: Training Coding Agents to Generalize Across Taskscited by 6
  27. 2025Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoningcited by 5
  28. 2025SynQuE: Estimating Synthetic Dataset Quality Without Annotationscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 3
  2. 2026Cowork-Gym: Synthesizing Interactive Work Environments at Scalecited by 0
  3. 2026Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Frontier-Eng: Benchmarking Self-Evolving Agents on Real-World Engineering Tasks with Generative Optimization
  2. 2026AutoEnvScaling: Automating the Data Flywheel with Terminal Agents
  3. 2025Solver-Informed RL: Grounding Large Language Models for Authentic Optimization Modeling
  4. 2026Yet Even Less Is Even Better For Agentic, Reasoning, and Coding LLMs
  5. 2026CodeMidas: Scaling Agentic Coding RL Environments from Code Itself
  6. 2026Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training
  7. 2026RLPF: Reinforcement Learning from Performance Feedback for Code Generation
  8. 2026Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills