mlsanityData to 14 Oct 2026

Optimal Transport Depth Up-Scaling

arXiv 2508.08011 ↗First posted 11 Aug 2025cs.CLcited by 4

Not on the map. Why?

Abstract (arXiv)

Pre-training Large Language Models (LLMs) from scratch at larger scales yields remarkable performance but incurs substantially high training costs. Depth up-scaling provides an efficient alternative by inserting new layers into a pre-trained LLM, avoiding training from scratch. However, most existing methods copying or averaging base layers for new layer, which misalign functionally corresponding neurons, leading to neuron permutation mismatch that harms performance. To address this issue, we propose Optimal Transport Depth Up-Scaling (OT-DUS), which leverages Optimal Transport (OT) theory to align and fuse functionally corresponding neurons module by module in adjacent base layers for new layer construction. OT-DUS achieves better overall performance in both general and specialized domains than existing methods for continual pre-training and supervised fine-tuning across different model sizes and model families. Our analysis of insertion strategies further finds that inserting new layers at higher positions yields not only stronger performance but also improved training efficiency.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 51 of its 74 references that are in the map. First posted on arXiv 11 Aug 2025.

Builds on

51 of 74 references.

References · 51

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  12. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
38 more
  1. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  2. 2020GLU Variants Improve Transformercited by 1,804
  3. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
  5. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,288
  6. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  7. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 1,009
  8. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 538
  9. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 384
  10. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 383
  11. 2021LexGLUE: A Benchmark Dataset for Legal Language Understanding in Englishcited by 310
  12. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
  13. 2015Net2Net: Accelerating Learning via Knowledge Transfercited by 250
  14. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 220
  15. 2019Model Fusion via Optimal Transportcited by 219
  16. 2019Neural Legal Judgment Prediction in Englishcited by 209
  17. 2021When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Datasetcited by 201
  18. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 195
  19. 2020The Computational Limits of Deep Learningcited by 193
  20. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  21. 2023Similarity of Neural Network Models: A Survey of Functional and Representational Measurescited by 131
  22. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  23. 2016Multifaceted Feature Visualization: Uncovering the Different Types of Features Learned By Each Neuron in Deep Neural Networkscited by 106
  24. 2024LLaMA Pro: Progressive LLaMA with Block Expansioncited by 101
  25. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  26. 2023Learning to Grow Pretrained Models for Efficient Transformer Trainingcited by 83
  27. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  28. 2016Network Morphismcited by 65
  29. 2023MultiLegalPile: A 689GB Multilingual Legal Corpuscited by 60
  30. 2017Multi-level Residual Networks from Dynamical Systems Viewcited by 59
  31. 2022Staged Training for Transformer Language Modelscited by 49
  32. 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Trainingcited by 43
  33. 2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
  34. 2023Masked Structural Growth for 2x Faster Language Model Pre-trainingcited by 33
  35. 2018Understanding Convolutional Neural Networks with Information Theory: An Initial Explorationcited by 32
  36. 2020Progressively Stacking 2.0: A Multi-stage Layerwise Training Method for BERT Training Speedupcited by 31
  37. 2019Statistical Model Aggregation via Parameter Matchingcited by 17
  38. 2024Investigating Low-Rank Training in Transformer Language Models: Efficiency and Scaling Analysiscited by 6

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026StateBridge: Training-free Hidden-state Alignment for Latent Communication in LLM Multi-Agent Systemscited by 5
  2. 2026Fundamental Reasoning Paradigms Induce Out-of-Domain Generalization in Language Modelscited by 3
  3. 2025MIDUS: Memory-Infused Depth Up-Scalingcited by 0
  4. 2026Adapting Text LLMs to Speech via Multimodal Depth Up-Scalingcited by 0