mlsanityData to 14 Oct 2026

LEMON: Lossless model expansion

arXiv 2310.07999 ↗First posted 12 Oct 2023cs.LG stat.MLcited by 23

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Scaling of deep neural networks, especially Transformers, is pivotal for their surging performance and has further led to the emergence of sophisticated reasoning capabilities in foundation models. Such scaling generally requires training large models from scratch with random initialization, failing to leverage the knowledge acquired by their smaller counterparts, which are already resource-intensive to obtain. To tackle this inefficiency, we present LosslEss MOdel ExpansioN (LEMON), a recipe to initialize scaled models using the weights of their smaller but pre-trained counterparts. This is followed by model training with an optimized learning rate scheduler tailored explicitly for the scaled models, substantially reducing the training time compared to training from scratch. Notably, LEMON is versatile, ensuring compatibility with various network structures, including models like Vision Transformers and BERT. Our empirical results demonstrate that LEMON reduces computational costs by 56.7% for Vision Transformers and 33.2% for BERT when compared to training from scratch.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 66 of its 84 references that are in the map. First posted on arXiv 12 Oct 2023.

Builds on

66 of 84 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  9. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,798
  4. 2016Neural Architecture Search with Reinforcement Learningcited by 1,499
  5. 2019Root Mean Square Layer Normalizationcited by 1,480
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  7. 2020On Layer Normalization in the Transformer Architecturecited by 922
  8. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  9. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 812
  10. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  11. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  12. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  13. 2016Learning Structured Sparsity in Deep Neural Networkscited by 541
  14. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
  15. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 479
  16. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  17. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 426
  18. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  19. 2016Designing Neural Network Architectures using Reinforcement Learningcited by 279
  20. 2016Exponential expressivity in deep neural networks through transient chaoscited by 261
  21. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  22. 2020Neural Architecture Search without Trainingcited by 244
  23. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  24. 2021Zero-Cost Proxies for Lightweight NAScited by 199
  25. 2018Dynamical Isometry and a Mean Field Theory of CNNs: How to Train 10,000-Layer Vanilla Convolutional Neural Networkscited by 176
  26. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  27. 2021Neural Architecture Search on ImageNet in Four GPU Hours: A Theoretically Inspired Perspectivecited by 169
  28. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  29. 2022Fine-tuned Language Models are Continual Learnerscited by 156
  30. 2022Signal Propagation in Transformers: Theoretical Perspectives and the Role of Rank Collapsecited by 141
  31. 2017Deep Rewiring: Training very sparse deep networkscited by 139
  32. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  33. 2022TimeLMs: Diachronic Language Models from Twittercited by 131
  34. 2019Evaluating the Search Phase of Neural Architecture Searchcited by 120
  35. 2021Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Trainingcited by 110
  36. 2019A Signal Propagation Perspective for Pruning Neural Networks at Initializationcited by 96
  37. 2021Rethinking Architecture Selection in Differentiable NAScited by 92
  38. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 5; 2026: 8.

23 papers.

Sort:

By area:

  1. 2025LongCat-Flash Technical Reportcited by 58
  2. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 43
  3. 2024Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initializationcited by 29
  4. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  5. 2024On the Inductive Bias of Stacking Towards Improving Reasoningcited by 19
  6. 2024A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMscited by 15
  7. 2024Efficient Stagewise Pretraining via Progressive Subnetworkscited by 10
  8. 2024DLO: Dynamic Layer Operation for Efficient Vertical Scaling of LLMscited by 10
  9. 2025Efficient Construction of Model Family through Progressive Training Using Model Expansioncited by 8
  10. 2024Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escape, and Network Embeddingcited by 6
  11. 2024Landscape-Aware Growing: The Power of a Little LAGcited by 4
  12. 2025PLM: Efficient Peripheral Language Models Hardware-Co-Designed for Ubiquitous Computingcited by 4
11 more
  1. 2026SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learningcited by 4
  2. 2026Exact Network Surgery: Functional Invariance and Gradient Plasticity in Reactive Computational Graphscited by 2
  3. 2025Scaling depth capacity via zero/one-layer model expansioncited by 1
  4. 2026Late-to-Early Training: LET LLMs Learn Earlier, So Faster and Bettercited by 1
  5. 2026Preservation Is Not Enough for Width Growth: Regime-Sensitive Selection of Dense LM Warm Startscited by 1
  6. 2024Gap Preserving Distillation by Building Bidirectional Mappings with A Dynamic Teachercited by 0
  7. 2025Advantageous Parameter Expansion Training Makes Better Large Language Modelscited by 0
  8. 2026CLIP-Map: Structured Matrix Mapping for Parameter-Efficient CLIP Compressioncited by 0
  9. 2026When is Warmstarting Effective for Scaling Language Models?cited by 0
  10. 2026What Does Model Growth Really Add? Functional Capacity Beyond Parameter Countcited by 0
  11. 2026Backdoor Learning in Language Models and Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Tiny Inference-Time Scaling with Latent Verifiers
  2. 2024Majority Kernels: An Approach to Leverage Big Model Dynamics for Efficient Small Model Training
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity
  4. 2024Phantom of Latent for Large Language and Vision Models
  5. 2024DεpS: Delayed ε-Shrinking for Faster Once-For-All Training
  6. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  7. 2026KITE: KV-Invariant Transformer Expansion for Efficient Agentic LLM Scaling
  8. 2025How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines