mlsanityData to 7 Oct 2026

LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning

arXiv 2606.24901 ↗First posted 12 Jun 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Continual learning capability is critical for Industrial LLMs, as deployed models must be continuously updated to meet evolving requirements and environments, rather than repeatedly retrained from scratch. However, most existing research focuses on improvements on static benchmarks, failing to capture real industrial needs. In this survey, we reformulate Industrial Continual Learning (ICL) for LLMs as a closed-loop update-and-release problem in a versioned ecosystem, where updates propagate hierarchically to industrial, application-specific models and LLM-powered applications, with capability inheritance and transfer across versions and model families. From this ecosystem perspective, we identify three core challenges: repeated adaptation erodes model plasticity, foundation-model upgrades break capability inheritance, and long-term sustainability is constrained by deployment requirements. We then organize the technical landscape of ICL around five lifecycle design principles: preserving plasticity headroom, treating upgrades as capability transfer, enabling trustworthy continual reinforcement learning, making training recipes self-optimizing, and building accountability as a base layer for long-term iteration. For each principle, we synthesize representative technical directions. Finally, we evaluate the maturity of each principle and its technical components via an evidence-based lens, identify key gaps hindering real-world deployment, and outline a practical ICL deployment blueprint and a pathway for feeding industrial realities back into academic research.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 82 of its 122 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

82 of 122 references.

References · 82

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2018Datasheets for Datasetscited by 1,371
  9. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  10. 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
38 more
  1. 2018Model Cards for Model Reportingcited by 1,068
  2. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
  3. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  4. 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
  5. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  6. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  7. 2025Muon is Scalable for LLM Trainingcited by 381
  8. 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
  9. 2019Orthogonal Gradient Descent for Continual Learningcited by 361
  10. 2021Gradient Projection Memory for Continual Learningcited by 338
  11. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  12. 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
  13. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  14. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  15. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  16. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  17. 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
  18. 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
  19. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  20. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  21. 2024CodeGemma: Open Code Models Based on Gemmacited by 145
  22. 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
  23. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
  24. 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
  25. 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
  26. 2024Med42-v2: A Suite of Clinical LLMscited by 89
  27. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  28. 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
  29. 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
  30. 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
  31. 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
  32. 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
  33. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  34. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  35. 2026Qwen3-Coder-Next Technical Reportcited by 60
  36. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  37. 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
  38. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42

50 most cited of 82 All 82 (JSON)

Built on by

Not cited yet.