mlsanityData to 14 Oct 2026

Superficial Self-Improved Reasoners Benefit from Model Merging

arXiv 2503.02103 ↗First posted 3 Mar 2025cs.CLcited by 16

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

As scaled language models (LMs) approach human-level reasoning capabilities, self-improvement emerges as a solution to synthesizing high-quality data corpus. While previous research has identified model collapse as a risk in self-improvement, where model outputs become increasingly deterministic, we discover a more fundamental challenge: the superficial self-improved reasoners phenomenon. In particular, our analysis reveals that even when LMs show improved in-domain (ID) reasoning accuracy, they actually compromise their generalized reasoning capabilities on out-of-domain (OOD) tasks due to memorization rather than genuine. Through a systematic investigation of LM architecture, we discover that during self-improvement, LM weight updates are concentrated in less reasoning-critical layers, leading to superficial learning. To address this, we propose Iterative Model Merging (IMM), a method that strategically combines weights from original and self-improved models to preserve generalization while incorporating genuine reasoning improvements. Our approach effectively mitigates both LM collapse and superficial learning, moving towards more stable self-improving systems.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 27 of its 78 references that are in the map. First posted on arXiv 3 Mar 2025.

Builds on

27 of 78 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024V-STaR: Training Verifiers for Self-Taught Reasoners9 Feb
  2. 2022Large Language Models Can Self-Improve20 Oct

Other references · 25

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2024Qwen2.5 Technical Reportcited by 4,942
  5. 2023Let's Verify Step by Stepcited by 3,789
  6. 2024OpenAI o1 System Cardcited by 1,244
  7. 2024Self-Rewarding Language Modelscited by 681
  8. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 662
  9. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 500
  10. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  11. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  12. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
13 more
  1. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  2. 2022Is synthetic data from generative models ready for image recognition?cited by 320
  3. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 204
  4. 2023On the Stability of Iterative Retraining of Generative Models on their own Datacited by 100
  5. 2023The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Textcited by 100
  6. 2024How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapsecited by 79
  7. 2024MathGenie: Generating Synthetic Data with Question Back-translation for Enhancing Mathematical Reasoning of LLMscited by 72
  8. 2024Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Samplingcited by 61
  9. 2024Self-Consistency Preference Optimizationcited by 50
  10. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  11. 2021A Closer Look at How Fine-tuning Changes BERTcited by 42
  12. 2024GameArena: Evaluating LLM Reasoning through Live Computer Gamescited by 32
  13. 2023Nepotistically Trained Generative-AI Models Collapsecited by 27

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 11; 2026: 5.

16 papers.

Sort:

By area:

  1. 2025Two Heads are Better Than One: Test-time Scaling of Multi-agent Collaborative Reasoningcited by 29
  2. 2025SwiReasoning: Switch-Thinking in Latent and Explicit for Pareto-Superior Reasoning LLMscited by 22
  3. 2025How to Protect Yourself from 5G Radiation? Investigating LLM Responses to Implicit Misinformationcited by 18
  4. 2025Mitigating Forgetting Between Supervised and Reinforcement Learning Yields Stronger Reasonerscited by 16
  5. 2026Behavior Knowledge Merge in Reinforced Agentic Modelscited by 11
  6. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  7. 2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
  8. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  9. 2025Assessing and Mitigating Medical Knowledge Drift and Conflicts in Large Language Modelscited by 4
  10. 2025Stabilizing Self-Consuming Diffusion Models with Latent Space Filteringcited by 4
  11. 2025Growing Through Experience: Scaling Episodic Grounding in Language Modelscited by 1
  12. 2025What Makes LLMs Effective Sequential Recommenders? A Study on Preference Intensity and Temporal Contextcited by 1
4 more
  1. 2026CopT: Contrastive On-Policy Thinking with Continuous Spaces for General and Agentic Reasoningcited by 1
  2. 2026Post-Hoc Merging is Not Enough: Many-Shot Model Merging with Loss-Gap Balancingcited by 1
  3. 2026CHORUS: Complementary Experts for High-Coverage Testbench Stimulus Generationcited by 0
  4. 2026The evolution of sex for artificial intelligence: a population-genetic framework for multigenerational model populationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Small Language Models Need Strong Verifiers to Self-Correct Reasoning
  2. 2025Self-Correction Bench: Uncovering and Addressing the Self-Correction Blind Spot in Large Language Models
  3. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  4. 2025Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models
  5. 2026Self-Improvement of Large Language Models: A Technical Overview and Future Outlook
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yet
  7. 2025The Thinking Spectrum: An Empirical Study of Tunable Reasoning in LLMs through Model Merging
  8. 2026More Yap Less Meaning: Uncovering Self-Improvement Behavior in SLMs