mlsanityData to 14 Oct 2026

ANCORA: Learning to Question via Manifold-Anchored Self-Play for Verifiable Reasoning

arXiv 2604.27644 ↗First posted 30 Apr 2026cs.LG cs.AI cs.PLcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose a paradigm shift toward open-ended curriculum self-play: rather than learning to answer on a fixed prompt set, a unified policy learns to question: generating verifiable problems, solving them, and turning verifier feedback into self-improvement without human-annotated solutions. We introduce ANCORA, in which the policy alternates between a Proposer that synthesizes novel specifications and a Solver that produces verified solutions, anchored by three load-bearing mechanisms: a two-level group-relative update coupling Proposer advantages across specifications with Solver advantages across solution attempts; iterative self-distilled SFT projecting the base model onto its valid-output manifold before RL; and a UCB-guided Curriculum DAG whose policy-induced problem set can provably expand under self-composition. Without these stabilizers, sparse verifier feedback drives Proposer collapse even under MLRL-aligned rewards; with them, ANCORA bootstraps a verifiable curriculum from zero human solutions. Instantiated in Verus, ANCORA lifts Dafny2Verus pass@1 from a 26.6% SFT baseline to 81.5% in test-time training (TTT, 0-shot), outperforming PSV self-play by 15.8 points despite PSV's 1-shot inference; in a transfer setting, training from Dafny2Verus seeds yields 36.2% and 17.2% pass@1 on held-out MBPP and HumanEval.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 52 references that are in the map. First posted on arXiv 30 Apr 2026.

Builds on

33 of 52 references.

References · 33

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2021Program Synthesis with Large Language Modelscited by 3,637
  6. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  7. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
  8. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  9. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  10. 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 815
  11. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  12. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
21 more
  1. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  2. 2025Absolute Zero: Reinforced Self-play Reasoning with Zero Datacited by 344
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  4. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 287
  5. 2025REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalizationcited by 241
  6. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  7. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 180
  8. 2025SPICE: Self-Play In Corpus Environments Improves Reasoningcited by 83
  9. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 79
  10. 2023Verus: Verifying Rust Programs using Linear Ghost Types (extended version)cited by 53
  11. 2024AutoVerus: Automated Proof Generation for Rust Codecited by 49
  12. 2026Dr. Zero: Self-Evolving Search Agents without Training Datacited by 44
  13. 2025Toward Training Superintelligent Software Agents through Self-Play SWE-RLcited by 40
  14. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinementcited by 35
  15. 2025Learning to Solve and Verify: A Self-Play Framework for Code and Test Generationcited by 35
  16. 2026Tool-R0: Self-Evolving LLM Agents for Tool-Learning from Zero Datacited by 34
  17. 2025On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimizationcited by 31
  18. 2025SPELL: Self-Play Reinforcement Learning for Evolving Long-Context Language Modelscited by 29
  19. 2025Propose, Solve, Verify: Self-Play Through Formal Verificationcited by 13
  20. 2026Teaching Models to Teach Themselves: Reasoning at the Edge of Learnabilitycited by 13
  21. 2026Optimizer-Induced Low-Dimensional Drift and Transverse Dynamics in Transformer Trainingcited by 4

Built on by

Not cited yet.