BOOST: BOttleneck-Optimized Scalable Training Framework for Low-Rank Large Language Models
Not on the map. Why?
Abstract (arXiv)
The scale of transformer model pre-training is constrained by the increasing computation and communication cost. Low-rank bottleneck architectures offer a promising solution to significantly reduce the training time and memory footprint with minimum impact on accuracy. Despite algorithmic efficiency, bottleneck architectures scale poorly under standard tensor parallelism. Simply applying 3D parallelism designed for full-rank methods leads to excessive communication and poor GPU utilization. To address this limitation, we propose BOOST, an efficient training framework tailored for large-scale low-rank bottleneck architectures. BOOST introduces a novel Bottleneck-aware Tensor Parallelism, and combines optimizations such as online-RMSNorm, linear layer grouping, and low-rank activation checkpointing to achieve end-to-end training speedup. Evaluations on different low-rank bottleneck architectures demonstrate that BOOST achieves 1.46-1.91× speedup over full-rank model baselines and 1.87-2.27× speedup over low-rank model with naively integrated 3D parallelism, with improved GPU utilization and reduced communication overhead.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 46 of its 63 references that are in the map. First posted on arXiv 13 Dec 2025.
Builds on
46 of 63 references.
References · 46
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2022Training Compute-Optimal Large Language Modelscited by 3,019
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
34 more
- 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
- 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
- 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
- 2025Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attentioncited by 407
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
- 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 265
- 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
- 2024SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compressioncited by 243
- 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
- 2022FP8 Formats for Deep Learningcited by 196
- 2018Mesh-TensorFlow: Deep Learning for Supercomputerscited by 169
- 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
- 2024TorchTitan: One-stop PyTorch native solution for production ready LLM pre-trainingcited by 85
- 2023FP8-LM: Training FP8 Large Language Modelscited by 77
- 2025Pretraining Large Language Models with NVFP4cited by 76
- 2023Training Transformers with 4-bit Integerscited by 60
- 2021Initialization and Regularization of Factorized Neural Layerscited by 58
- 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 54
- 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
- 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 45
- 2023Zero Bubble Pipeline Parallelismcited by 40
- 2025FP4 All the Way: Fully Quantized Training of LLMscited by 40
- 2025MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Productioncited by 31
- 2022Exploring Low Rank Training of Deep Neural Networkscited by 30
- 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
- 2025MoE Parallel Folding: Heterogeneous Parallelism Mappings for Efficient Large-Scale MoE Model Training with Megatron Corecited by 17
- 2025A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Modelscited by 16
- 2024CoMERA: Computing- and Memory-Efficient Training via Rank-Adaptive Tensor Optimizationcited by 15
- 2024CompAct: Compressed Activations for Memory-Efficient LLM Trainingcited by 15
- 2024LoQT: Low-Rank Adapters for Quantized Pretrainingcited by 12
- 2025LOST: Low-rank and Sparse Pre-training for Large Language Modelscited by 10
- 2025CR-Net: Scaling Parameter-Efficient Training with Cross-Layer Low-Rank Structurecited by 7
- 2025LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossingcited by 6
Built on by
Citing papers per year: 2026: 2.
2 papers.