LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning
Not on the map. Why?
Abstract (arXiv)
Continual learning capability is critical for Industrial LLMs, as deployed models must be continuously updated to meet evolving requirements and environments, rather than repeatedly retrained from scratch. However, most existing research focuses on improvements on static benchmarks, failing to capture real industrial needs. In this survey, we reformulate Industrial Continual Learning (ICL) for LLMs as a closed-loop update-and-release problem in a versioned ecosystem, where updates propagate hierarchically to industrial, application-specific models and LLM-powered applications, with capability inheritance and transfer across versions and model families. From this ecosystem perspective, we identify three core challenges: repeated adaptation erodes model plasticity, foundation-model upgrades break capability inheritance, and long-term sustainability is constrained by deployment requirements. We then organize the technical landscape of ICL around five lifecycle design principles: preserving plasticity headroom, treating upgrades as capability transfer, enabling trustworthy continual reinforcement learning, making training recipes self-optimizing, and building accountability as a base layer for long-term iteration. For each principle, we synthesize representative technical directions. Finally, we evaluate the maturity of each principle and its technical components via an evidence-based lens, identify key gaps hindering real-world deployment, and outline a practical ICL deployment blueprint and a pathway for feeding industrial realities back into academic research.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 82 of its 122 references that are in the map. First posted on arXiv 12 Jun 2026.
Builds on
82 of 122 references.
References · 82
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2025Qwen3 Technical Reportcited by 8,195
- 2020Scaling Laws for Neural Language Modelscited by 6,411
- 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
- 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
- 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2018Datasheets for Datasetscited by 1,371
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,276
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
38 more
- 2018Model Cards for Model Reportingcited by 1,068
- 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 895
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
- 2025GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Modelscited by 446
- 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
- 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
- 2025Muon is Scalable for LLM Trainingcited by 381
- 2019Online Continual Learning with Maximally Interfered Retrievalcited by 376
- 2019Orthogonal Gradient Descent for Continual Learningcited by 361
- 2021Gradient Projection Memory for Continual Learningcited by 338
- 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
- 2021Rainbow Memory: Continual Learning with a Memory of Diverse Samplescited by 273
- 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
- 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
- 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
- 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
- 2024RegMix: Data Mixture as Regression for Language Model Pre-trainingcited by 172
- 2025RL's Razor: Why Online Reinforcement Learning Forgets Lesscited by 162
- 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
- 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
- 2024CodeGemma: Open Code Models Based on Gemmacited by 145
- 2015Adding Gradient Noise Improves Learning for Very Deep Networkscited by 144
- 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 122
- 2024Mitigating Catastrophic Forgetting in Large Language Models with Self-Synthesized Rehearsalcited by 114
- 2020Memory-Efficient Incremental Learning Through Feature Adaptationcited by 107
- 2024Med42-v2: A Suite of Clinical LLMscited by 89
- 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
- 2025Fin-R1: A Large Language Model for Financial Reasoning through Reinforcement Learningcited by 82
- 2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
- 2022TRGP: Trust Region Gradient Projection for Continual Learningcited by 79
- 2023Using Large Language Models for Hyperparameter Optimizationcited by 71
- 2025Stabilizing MoE Reinforcement Learning by Aligning Training and Inference Routerscited by 67
- 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
- 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
- 2026Qwen3-Coder-Next Technical Reportcited by 60
- 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
- 2025Seed-Coder: Let the Code Model Curate Data for Itselfcited by 46
- 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
50 most cited of 82 All 82 (JSON)
Built on by
Not cited yet.