Beyond Reasoning Gains: Mitigating General-Capability Forgetting in Large Reasoning Models
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement learning with verifiable rewards (RLVR) has delivered impressive gains in mathematical and multimodal reasoning and has become a standard post-training paradigm for contemporary language and vision-language models. However, the RLVR recipe introduces a significant risk of capability regression, in which models forget foundational skills after prolonged training without employing regularization strategies. We empirically confirm this concern, observing that open-source reasoning models suffer performance degradation on core capabilities such as perception and faithfulness. While imposing regularization terms like KL divergence can help prevent deviation from the base model, these terms are computed on the current task and therefore do not guarantee preservation of broader knowledge. Meanwhile, commonly used experience replay across heterogeneous domains makes it nontrivial to decide how much training emphasis each objective should receive. To address this, we propose RECAP-a replay strategy with dynamic objective reweighting for general knowledge preservation. Our reweighting mechanism adapts online using short-horizon signals of convergence and instability, shifting the post-training focus away from saturated objectives and toward underperforming or volatile ones. Our method is end-to-end and readily applicable to existing RLVR pipelines without training additional models or heavy tuning. Extensive experiments on benchmarks using Qwen2.5-VL-3B and Qwen2.5-VL-7B demonstrate the effectiveness of our method, which not only preserves general capabilities but also improves reasoning by enabling more flexible trade-offs among in-task rewards.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 101 of its 123 references that are in the map. First posted on arXiv 24 Oct 2025.
Builds on
101 of 123 references.
Closest ancestors · 16
Same direction, newest first
- 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
- 2025RLEP: Reinforcement Learning with Experience Replay for LLM Reasoning10 Jul
- 2025SRFT: A Single-Stage Method with Supervised and Reinforcement Fine-Tuning for Reasoning24 Jun
- 2025ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models30 May
- 2025Maximizing Confidence Alone Improves Reasoning28 May
- 2025Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models15 May
- 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?18 Apr
- 2025Open-Reasoner-Zero: An Open Source Approach to Scaling Up Reinforcement Learning on the Base Model31 Mar
- 2025Understanding R1-Zero-Like Training: A Critical Perspective26 Mar
- 2025Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models16 Mar
- 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learning20 Feb
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
4 more
Other references · 85
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
38 more
- 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
- 2020Learning to summarize from human feedbackcited by 2,588
- 2016Learning without Forgettingcited by 2,496
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
- 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2017Gradient Episodic Memory for Continual Learningcited by 1,898
- 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
- 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2017Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semanticscited by 1,358
- 2020Gradient Surgery for Multi-Task Learningcited by 1,256
- 2017Mixed Precision Trainingcited by 1,156
- 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,105
- 2018Experience Replay for Continual Learningcited by 1,095
- 2018Efficient Lifelong Learning with A-GEMcited by 1,058
- 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 911
- 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
- 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 834
- 2016A Diagram Is Worth A Dozen Imagescited by 804
- 2023LISA: Reasoning Segmentation via Large Language Modelcited by 776
- 2017Deep Reinforcement Learning that Matterscited by 754
- 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
- 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
- 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 586
- 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
- 2018End-to-End Multi-Task Learning with Attentioncited by 534
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 526
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
- 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 476
- 2021Conflict-Averse Gradient Descent for Multi-task Learningcited by 459
- 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
- 2025Reasoning Models Don't Always Say What They Thinkcited by 415
- 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 400
- 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 353
- 2025VL-Rethinker: Incentivizing Self-Reflection of Vision-Language Models with Reinforcement Learningcited by 323
50 most cited of 85 All 85 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 4.
5 papers.
- 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgerycited by 9
- 2025Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMscited by 4
- 2026Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFTcited by 1
- 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVRcited by 1
- 2026Constraint-Rectified Training for Efficient Chain-of-Thoughtcited by 0
- 2026Learning to Solve, Forgetting to Retain: Correct-Set Turnover in RLVR2 Jun 2026
- 2026Preserving Foundational Capabilities in Flow-Matching VLAs through Conservative SFT9 May 2026
- 2026Constraint-Rectified Training for Efficient Chain-of-Thought13 Feb 2026
- 2026Advancing General-Purpose Reasoning Models with Modular Gradient Surgery2 Feb 2026
- 2025Reinforcement Learning Improves Traversal of Parametric Knowledge in LLMs8 Nov 2025
Similar, not cited
Close in wording, no citation either way
- 2025The Debate on RLVR Reasoning Capability Boundary: Shrinkage, Expansion, or Both? A Two-Stage Dynamic View
- 2026Resource-Efficient Reinforcement for Reasoning Large Language Models via Dynamic One-Shot Policy Refinement
- 2025Boosting Reasoning in Large Multimodal Models via Activation Replay
- 2026Training Reasoning Models on Saturated Problems via Failure-Prefix Conditioning
- 2026SAGE: Shaping Anchors for Guided Exploration in RLVR of LLMs
- 2025The Invisible Leash: Why RLVR May or May Not Escape Its Origin
- 2025Extending RLVR to Open-Ended Tasks via Verifiable Multiple-Choice Reformulation
- 2025The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models