mlsanityData to 7 Oct 2026

PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matrices

arXiv 2211.14133 ↗First posted 25 Nov 2022cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Pipeline parallelism enables efficient training of Large Language Models (LLMs) on large-scale distributed accelerator clusters. Yet, pipeline bubbles during startup and tear-down reduce the utilization of accelerators. Although efficient pipeline schemes with micro-batching and bidirectional pipelines have been proposed to maximize utilization, a significant number of bubbles cannot be filled using synchronous forward and backward passes. To address this problem, we suggest that extra work be assigned to the bubbles to gain auxiliary benefits in LLM training. As an example in this direction, we propose PipeFisher, which assigns the work of K-FAC, a second-order optimization method based on the Fisher information matrix, to the bubbles to accelerate convergence. In Phase 1 pretraining of BERT-Base and -Large models, PipeFisher reduces the (simulated) training time to 50-75% compared to training with a first-order optimizer by greatly improving the accelerator utilization and benefiting from the improved convergence by K-FAC.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 25 Nov 2022.

Builds on

24 of 39 references.

References · 24

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  8. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  9. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  12. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
12 more
  1. 2019Rigging the Lottery: Making All Tickets Winnerscited by 491
  2. 2014New insights and perspectives on the natural gradient methodcited by 435
  3. 2018Shampoo: Preconditioned Stochastic Tensor Optimizationcited by 413
  4. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  5. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  6. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  7. 2020Continual Deep Learning by Functional Regularisation of Memorable Pastcited by 104
  8. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  9. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  10. 2020Convolutional Neural Network Training with Distributed K-FACcited by 22
  11. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  12. 2020Scalable and Practical Natural Gradient for Large-Scale Deep Learningcited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 8; 2025: 4; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 84
  2. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  3. 2023Kronecker-Factored Approximate Curvature for Modern Neural Network Architecturescited by 49
  4. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  5. 2024Influence Functions for Scalable Data Attribution in Diffusion Modelscited by 31
  6. 2023Reducing Energy Bloat in Large Model Trainingcited by 22
  7. 2024Kronecker-Factored Approximate Curvature for Physics-Informed Neural Networkscited by 21
  8. 2025Purifying Shampoo: Investigating Shampoo's Heuristics by Decomposing its Preconditionercited by 21
  9. 2023The Geometry of Neural Nets' Parameter Spaces Under Reparametrizationcited by 16
  10. 2023On the Parameterization of Second-Order Optimization Effective Towards the Infinite Widthcited by 14
  11. 2024Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitationcited by 14
  12. 2023Structured Inverse-Free Natural Gradient: Memory-Efficient & Numerically-Stable KFACcited by 9
12 more
  1. 2024Fisher Information-based Efficient Curriculum Federated Learning with Large Language Modelscited by 6
  2. 2024PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Trainingcited by 5
  3. 2025A High-Dimensional Statistical Method for Optimizing Transfer Quantities in Multi-Source Transfer Learningcited by 5
  4. 2023PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Predictioncited by 4
  5. 2024Improving training time and GPU utilization in geo-distributed language model trainingcited by 3
  6. 2022PiPar: Pipeline Parallelism for Collaborative Machine Learningcited by 2
  7. 2023Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorizationcited by 2
  8. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 2
  9. 2026Unified Optimization of Source Weights and Transfer Quantities in Multi-Source Transfer Learning: An Asymptotic Frameworkcited by 1
  10. 2026Layerwise LQR for Geometry-Aware Optimization of Deep Networkscited by 1
  11. 2024FreeRide: Harvesting Bubbles in Pipeline Parallelismcited by 0
  12. 2025GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Modelscited by 0