Data to 14 Oct 2026

AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy

arXiv 2506.13284 ↗First posted 16 Jun 2025cs.CL cs.AI cs.LGcited by 77

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

In this work, we investigate the synergy between supervised fine-tuning (SFT) and reinforcement learning (RL) in developing strong reasoning models. We begin by curating the SFT training data through two scaling strategies: increasing the number of collected prompts and the number of generated responses per prompt. Both approaches yield notable improvements in reasoning performance, with scaling the number of prompts resulting in more substantial gains. We then explore the following questions regarding the synergy between SFT and RL: (i) Does a stronger SFT model consistently lead to better final performance after large-scale RL training? (ii) How can we determine an appropriate sampling temperature during RL training to effectively balance exploration and exploitation for a given SFT initialization? Our findings suggest that (i) holds true, provided effective RL training is conducted, particularly when the sampling temperature is carefully chosen to maintain the temperature-adjusted entropy around 0.3, a setting that strikes a good balance between exploration and exploitation. Notably, the performance gap between initial SFT models narrows significantly throughout the RL process. Leveraging a strong SFT foundation and insights into the synergistic interplay between SFT and RL, our AceReason-Nemotron-1.1 7B model significantly outperforms AceReason-Nemotron-1.0 and achieves new state-of-the-art performance among Qwen2.5-7B-based reasoning models on challenging math and code benchmarks, thereby demonstrating the effectiveness of our post-training recipe. We release the model and data at: https://huggingface.co/nvidia/AceReason-Nemotron-1.1-7B

Key facts. Cited by 77 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 33 of its 42 references that are in the map. First posted on arXiv 16 Jun 2025.

Builds on

33 of 42 references.

Closest ancestors · 5

Same direction, newest first

Other references · 28

2024The Llama 3 Herd of Modelscited by 14,715
2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
2021Training Verifiers to Solve Math Word Problemscited by 10,109
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
2025Qwen3 Technical Reportcited by 8,820
2021Evaluating Large Language Models Trained on Codecited by 8,320
2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
2024Qwen2.5 Technical Reportcited by 4,942
2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391

16 more

2025s1: Simple test-time scalingcited by 1,341
2024Qwen2.5-Coder Technical Reportcited by 1,203
2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
2021Measuring Coding Challenge Competence With APPScited by 902
2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
2025Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyondcited by 175
2025AIMO-2 Winning Solution: Building State-of-the-Art Mathematical Reasoning Models with OpenMathReasoning datasetcited by 174
2025Skywork Open Reasoner 1 Technical Reportcited by 165
2023TACO: Topics in Algorithmic COde generation datasetcited by 160
2025Llama-Nemotron: Efficient Reasoning Modelscited by 158
2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 133
2024OpenCoder: The Open Cookbook for Top-Tier Code Large Language Modelscited by 130
2024NVLM: Open Frontier-Class Multimodal LLMscited by 125
2025OpenCodeReasoning: Advancing Data Distillation for Competitive Codingcited by 113
2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 77
2024Evaluating Language Models for Efficient Code Generationcited by 73

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 38; 2026: 39.

77 papers.

Sort:

By area:

2025DiffuCoder: Understanding and Improving Masked Diffusion Models for Code Generationcited by 187
2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 180
2025Scaling Latent Reasoning via Looped Language Modelscited by 165
2025SimpleVLA-RL: Scaling VLA Training via Reinforcement Learningcited by 151
2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
2025On-Policy RL Meets Off-Policy Experts: Harmonizing Supervised Fine-Tuning and Reinforcement Learning via Dynamic Weightingcited by 91
2025rStar2-Agent: Agentic Reasoning Technical Reportcited by 54
2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
2025Klear-Reasoner: Advancing Reasoning Capability via Gradient-Preserving Clipping Policy Optimizationcited by 39
2025The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Rewardcited by 37
2025RL Squeezes, SFT Expands: A Comparative Study of Reasoning LLMscited by 34

38 more

2025Single-stream Policy Optimizationcited by 28
2025Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Insteadcited by 28
2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
2025Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Datacited by 27
2026Quantization-Aware Distillation for NVFP4 Inference Accuracy Recoverycited by 22
2026Composer 2 Technical Reportcited by 22
2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
2025Don't Pass@k: A Bayesian Framework for Large Language Model Evaluationcited by 15
2026Memory-Efficient Looped Transformer: Decoupling Compute from Memory in Looped Language Modelscited by 14
2025HoneyBee: Data Recipes for Vision-Language Reasonerscited by 13
2025The Synergy Dilemma of Long-CoT SFT and RL: Investigating Post-Training Techniques for Reasoning VLMscited by 12
2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
2025Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoningcited by 11
2025K2-Think: A Parameter-Efficient Reasoning Systemcited by 11
2025Scaling Generalist Data-Analytic Agentscited by 11
2026Entropy-Preserving Supervised Fine-Tuning via Adaptive Self-Distillation for Large Reasoning Modelscited by 9
2025The Signal is in the Steps: Local Scoring for Reasoning Data Selectioncited by 8
2025Towards Understanding Self-play for LLM Reasoningcited by 8
2026Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learningcited by 8
2026How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Datacited by 8
2026Loop the Loopies!cited by 8
2025FAPO: Flawed-Aware Policy Optimization for Efficient and Reliable Reasoningcited by 7
2025ScaleDiff: Scaling Difficult Problems for Advanced Mathematical Reasoningcited by 6
2025Where Did This Sentence Come From? Tracing Provenance in LLM Reasoning Distillationcited by 6
2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-trainingcited by 6
2026Statistical Early Stopping for Reasoning Modelscited by 6
2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 6
2025Audio Flamingo Sound-CoT Technical Report: Improving Chain-of-Thought Reasoning in Sound Understandingcited by 5
2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapsecited by 5
2025Thinking Augmented Pre-trainingcited by 4
2026Logics-STEM: Empowering LLM Reasoning via Failure-Driven Post-Training and Document Knowledge Enhancementcited by 4
2026From SFT to RL: Demystifying the Post-Training Pipeline for LLM-based Vulnerability Detectioncited by 4
2026Post-Trained MoE Can Skip Half Experts via Self-Distillationcited by 4
2026Credit Assignment with Resets in Language Model Reasoningcited by 4
2025More Than One Teacher: Adaptive Multi-Guidance Policy Optimization for Diverse Explorationcited by 3
2026MapTab: A Diagnostic Benchmark for Long-Horizon Multi-Criteria Multimodal Reasoning on Heterogeneous Topological Graphscited by 3
2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMscited by 3
2025Video Reasoning without Trainingcited by 2

50 most cited of 77 All 77 (JSON)

Similar, not cited

Close in wording, no citation either way