mlsanityData to 7 Oct 2026

Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelines

arXiv 2107.06925 ↗First posted 14 Jul 2021cs.DC cs.LGcited by 68

Not on the map. Why?

Abstract (arXiv)

Training large deep learning models at scale is very challenging. This paper proposes Chimera, a novel pipeline parallelism scheme which combines bidirectional pipelines for efficiently training large-scale models. Chimera is a synchronous approach and therefore no loss of accuracy, which is more convergence-friendly than asynchronous approaches. Compared with the latest synchronous pipeline approach, Chimera reduces the number of bubbles by up to 50%; benefiting from the sophisticated scheduling of bidirectional pipelines, Chimera has a more balanced activation memory consumption. Evaluations are conducted on Transformer based language models. For a GPT-2 model with 1.3 billion parameters running on 2,048 GPU nodes of the Piz Daint supercomputer, Chimera improves the training throughput by 1.16x-2.34x over the state-of-the-art synchronous and asynchronous pipeline approaches.

Key facts. Cited by 68 papers in the map (data through 7 Oct 2026). Builds on 33 of its 59 references that are in the map. First posted on arXiv 14 Jul 2021.

Builds on

33 of 59 references.

References · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2020End-to-End Object Detection with Transformerscited by 7,384
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  11. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  12. 2018Regularized Evolution for Image Classifier Architecture Searchcited by 902
21 more
  1. 2020Pre-Trained Image Processing Transformercited by 891
  2. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  3. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 671
  4. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  5. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 339
  6. 2018Horovod: fast and easy distributed deep learning in TensorFlowcited by 275
  7. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  8. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 177
  9. 2017Asynchronous Decentralized Parallel Stochastic Gradient Descentcited by 159
  10. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  11. 2018Stochastic Gradient Push for Distributed Deep Learningcited by 152
  12. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  13. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  14. 2019PipeMare: Asynchronous Pipeline Parallel DNN Trainingcited by 54
  15. 2018SparCML: High-Performance Sparse Communication for Machine Learningcited by 34
  16. 2019Large-Batch Training for LSTM and Beyondcited by 21
  17. 2019Red-blue pebbling revisited: near optimal parallel matrix-matrix multiplicationcited by 21
  18. 2020An In-Depth Analysis of the Slingshot Interconnectcited by 13
  19. 2017AMPNet: Asynchronous Model-Parallel Training for Dynamic Neural Networkscited by 12
  20. 2019Taming Unbalanced Training Workloads in Deep Learning with Partial Collective Operationscited by 9
  21. 2020Breaking (Global) Barriers in Parallel Stochastic Optimization with Wait-Avoiding Group Averagingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 10; 2023: 15; 2024: 18; 2025: 11; 2026: 12.

68 papers.

Sort:

By area:

  1. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  2. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  3. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  4. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  5. 2024ExeGPT: Constraint-Aware Resource Scheduling for LLM Inferencecited by 43
  6. 2024Optimizing RLHF Training for Large Language Models with Stage Fusioncited by 41
  7. 2023Oobleck: Resilient Distributed Training of Large Models Using Pipeline Templatescited by 40
  8. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuningcited by 39
  9. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  10. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  11. 2022Near-Optimal Sparse Allreduce for Distributed Deep Learningcited by 23
  12. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
38 more
  1. 2026Scalable Training of Mixture-of-Experts Models with Megatron Corecited by 16
  2. 2023Optimus-CC: Efficient Large NLP Model Training with 3D Parallelism Aware Communication Compressioncited by 14
  3. 2024ReCycle: Resilient Training of Large DNNs using Pipeline Adaptationcited by 14
  4. 2024Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitationcited by 14
  5. 2025Expert-as-a-Service: Towards Efficient, Scalable, and Robust Large-scale MoE Servingcited by 13
  6. 2023Optimizing Distributed Training on Frontier for Large Language Modelscited by 12
  7. 2022Merak: An Efficient Distributed DNN Training Framework with Automated 3D Parallelism for Giant Foundation Modelscited by 11
  8. 2024Training and Serving System of Foundation Models: A Comprehensive Surveycited by 11
  9. 2024Pipeline Parallelism with Controllable Memorycited by 11
  10. 2024Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Surveycited by 9
  11. 2024GraphPipe: Improving Performance and Scalability of DNN Training with Graph Pipeline Parallelismcited by 9
  12. 2022HammingMesh: A Network Topology for Large-Scale Deep Learningcited by 8
  13. 2026UltraEP: Unleash MoE Training and Inference on Rack-Scale Nodes with Near-Optimal Load Balancingcited by 8
  14. 2024Hardware Scaling Trends and Diminishing Returns in Large-Scale Distributed Trainingcited by 7
  15. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5
  16. 2023Colossal-Auto: Unified Automation of Parallelization and Activation Checkpoint for Large-scale Modelscited by 5
  17. 2024PipeFill: Using GPUs During Bubbles in Pipeline-parallel LLM Trainingcited by 5
  18. 2022Survey on Large Scale Neural Network Trainingcited by 4
  19. 2023Systems for Parallel and Distributed Large-Model Deep Learning Trainingcited by 4
  20. 2023GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelismcited by 4
  21. 2023PipeOptim: Ensuring Effective 1F1B Schedule with Optimizer-Dependent Weight Predictioncited by 4
  22. 2024From promise to practice: realizing high-performance decentralized trainingcited by 4
  23. 2021Hydra: A System for Large Multi-Model Deep Learningcited by 3
  24. 2024Sparse Checkpointing for Fast and Reliable MoE Trainingcited by 3
  25. 2025DIP: Efficient Large Multimodal Model Training with Dynamic Interleaved Pipelinecited by 3
  26. 2022Elixir: Train a Large Language Model on a Small GPU Clustercited by 2
  27. 2022PiPar: Pipeline Parallelism for Collaborative Machine Learningcited by 2
  28. 2023UniAP: Unifying Inter- and Intra-Layer Automatic Parallelism by Mixed Integer Quadratic Programmingcited by 2
  29. 2023Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Searchcited by 2
  30. 2024HETHUB: A Distributed Training System with Heterogeneous Cluster for Large-Scale Modelscited by 2
  31. 2024Resource Allocation and Workload Scheduling for Large-Scale Distributed Deep Learning: A Surveycited by 2
  32. 2024Optimizing Large Model Training through Overlapped Activation Recomputationcited by 2
  33. 2025HelixPipe: Efficient Distributed Training of Long Sequence Transformers with Attention Parallel Pipeline Parallelismcited by 2
  34. 2025LeMix: Unified Scheduling for LLM Training and Inference on Multi-GPU Systemscited by 2
  35. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  36. 2026Mitigating Staleness in Asynchronous Pipeline Parallelism via Basis Rotationcited by 2
  37. 2026TideRL: Boosting Agentic RL Goodput with Readiness-Aware Schedulingcited by 2
  38. 2023ADA-GP: Accelerating DNN Training By Adaptive Gradient Predictioncited by 1

50 most cited of 68 All 68 (JSON)