SUPERNOVA: Eliciting General Reasoning in LLMs with Reinforcement Learning on Natural Instructions
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement Learning with Verifiable Rewards (RLVR) has substantially improved reasoning in formal domains such as mathematics and code, but extending these gains beyond STEM remains challenging. Extending RLVR beyond STEM is fundamentally constrained by the lack of high-quality verifiable training data. In this work, we introduce SUPERNOVA, a framework for curating RLVR data from natural instruction datasets, which are a rich source of expert-annotated data but are underexplored for RLVR training. Through 100+ controlled RL experiments, we systematically study how to utilize these dataset for RLVR and how data curation decisions affect downstream reasoning performance . In particular, we investigate three data designs: (a) source task selection, (b) task mixing, and (c) synthetic interventions. Our analysis reveals that source task selection has a significant impact on downstream reasoning performance. Moreover, selecting tasks based on their performance for individual target tasks outperforms strategies based on overall average performance and synthetic interventions do not improve reasoning. Guided by these insights, we construct SUPERNOVA, a high-quality RLVR dataset of 25K instances curated from natural instruction datasets. We show that training Qwen3-0.6B on SUPERNOVA outperforms the base Qwen3-0.6B, yielding a relative gain of 64.4pp on BigBench Extra Hard (BBEH), a challenging benchmark comprising 23 complex reasoning tasks. Importantly, we find that gains from SUPERNOVA generalize to unseen benchmarks, larger model scales, and newer model families. Overall, our findings provide practical insights for curating human-annotated resources to extend RLVR to general reasoning. Models, Data, Code at https://github.com/asuvarna31/supernova.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 38 of its 49 references that are in the map. First posted on arXiv 9 Apr 2026.
Builds on
38 of 49 references.
Closest ancestors · 14
Same direction, newest first
- 2026Golden Goose: A Simple Trick to Synthesize Unlimited RLVR Tasks from Unverifiable Internet Text30 Jan
- 2025Language Models that Think, Chat Better24 Sep
- 2025Revisiting Reinforcement Learning for LLM Reasoning from A Cross-Domain Perspective17 Jun
- 2025RuleReasoner: Reinforced Rule-based Reasoning via Domain-aware Dynamic Sampling10 Jun
- 2025SynLogic: Synthesizing Verifiable Reasoning Data at Scale for Learning Logical Reasoning and Beyond26 May
- 2025AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning22 May
- 2025General-Reasoner: Advancing LLM Reasoning Across All Domains20 May
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
- 2025Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning15 Apr
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
- 2025SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild24 Mar
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
2 more
Other references · 24
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen3 Technical Reportcited by 8,195
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,203
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2025s1: Simple test-time scalingcited by 1,311
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
- 2023Instruction Tuning for Large Language Models: A Surveycited by 617
- 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 514
12 more
- 2025LIMO: Less is More for Reasoningcited by 488
- 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
- 2025ZebraLogic: On the Scaling Limits of LLMs for Logical Reasoningcited by 186
- 2025BIG-Bench Extra Hardcited by 123
- 2024Large Language Models Can Learn Temporal Reasoningcited by 110
- 2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 106
- 2025Does Math Reasoning Improve General LLM Capabilities? Understanding Transferability of LLM Reasoningcited by 101
- 20251.4 Million Open-Source Distilled Reasoning Dataset to Empower Large Language Model Trainingcited by 49
- 2020Understanding Human Intelligence through Human Limitationscited by 33
- 2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
- 2024ROSE: A Reward-Oriented Data Selection Framework for LLM Task-Specific Instruction Tuningcited by 10
- 2025Does Learning Mathematical Problem-Solving Generalize to Broader Reasoning?cited by 7
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMs
- 2025RLPR: Extrapolating RLVR to General Domains without Verifiers
- 2025Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning
- 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
- 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
- 2026From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement
- 2025Reinforcement Learning vs. Distillation: Understanding Accuracy and Capability in LLM Reasoning
- 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Following