mlsanityData to 14 Oct 2026

Towards a Flexible and High-Fidelity Approach to Distributed DNN Training Emulation

arXiv 2405.02969 ↗First posted 5 May 2024cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

We propose NeuronaBox, a flexible, user-friendly, and high-fidelity approach to emulate DNN training workloads. We argue that to accurately observe performance, it is possible to execute the training workload on a subset of real nodes and emulate the networked execution environment along with the collective communication operations. Initial results from a proof-of-concept implementation show that NeuronaBox replicates the behavior of actual systems with high accuracy, with an error margin of less than 1% between the emulated measurements and the real system.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 5 May 2024.

Builds on

18 of 37 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  7. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  9. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 478
  10. 2019Selection via Proxy: Efficient Data Selection for Deep Learningcited by 339
  11. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  12. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 201
6 more
  1. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  2. 2013Petuum: A New Platform for Distributed Machine Learning on Big Datacited by 35
  3. 2023vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Trainingcited by 23
  4. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 16
  5. 2018Revisiting Network Support for RDMAcited by 8
  6. 2022DeepFlow: A Cross-Stack Pathfinding Framework for Distributed AI Systemscited by 6

Built on by

Not cited yet.