mlsanityData to 7 Oct 2026

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

arXiv 2408.14158 ↗First posted 26 Aug 2024cs.DC cs.AIcited by 8

Not on the map. Why?

Abstract (arXiv)

The rapid progress in Deep Learning (DL) and Large Language Models (LLMs) has exponentially increased demands of computational power and bandwidth. This, combined with the high costs of faster computing chips and interconnects, has significantly inflated High Performance Computing (HPC) construction costs. To address these challenges, we introduce the Fire-Flyer AI-HPC architecture, a synergistic hardware-software co-design framework and its best practices. For DL training, we deployed the Fire-Flyer 2 with 10,000 PCIe A100 GPUs, achieved performance approximating the DGX-A100 while reducing costs by half and energy consumption by 40%. We specifically engineered HFReduce to accelerate allreduce communication and implemented numerous measures to keep our Computation-Storage Integrated Network congestion-free. Through our software stack, including HaiScale, 3FS, and HAI-Platform, we achieved substantial scalability by overlapping computation and communication. Our system-oriented experience from DL training provides valuable insights to drive future advancements in AI-HPC.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 30 of its 106 references that are in the map. First posted on arXiv 26 Aug 2024.

Builds on

30 of 106 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2023GPT-4 Technical Reportcited by 18,132
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  8. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
18 more
  1. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  2. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  3. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  4. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  5. 2024DeepSeek LLM: Scaling Open-Source Language Models with Longtermismcited by 621
  6. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  7. 2024DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligencecited by 325
  8. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  9. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  10. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  11. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  12. 2024MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUscited by 111
  13. 2024AI and Memory Wallcited by 89
  14. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  15. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  16. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  17. 2023Zero Bubble Pipeline Parallelismcited by 39
  18. 2023Benchmarking and In-depth Performance Study of Large Language Models on Habana Gaudi Processorscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 4.

8 papers.

  1. 2025Inference-Time Scaling for Generalist Reward Modelingcited by 233
  2. 2025Decentralized Diffusion Modelscited by 12
  3. 2025The Energy-Efficient Hierarchical Neural Network with Fast FPGA-Based Incremental Learningcited by 1
  4. 2025BootSeer: Analyzing and Mitigating Initialization Bottlenecks in Large-Scale LLM Trainingcited by 0
  5. 2026Joint Hardware-Workload Co-Optimization for In-Memory Computing Acceleratorscited by 0
  6. 2026HetCCL: Enabling Collective Communication For Mixed-Vendor Heterogeneous Clusterscited by 0
  7. 2026OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetchingcited by 0
  8. 2026Efficient Expert-Parallel Communication on PCIe-Connected Consumer GPUscited by 0