mlsanityData to 14 Oct 2026

NEST: Network- and Memory-Aware Device Placement For Distributed Deep Learning

arXiv 2603.06798 ↗First posted 6 Mar 2026cs.LG cs.DC stat.MLcited by 0

Not on the map. Why?

Abstract (arXiv)

The growing scale of deep learning demands distributed training frameworks that jointly reason about parallelism, memory, and network topology. Prior works often rely on heuristic or topology-agnostic search, handling communication and memory separately. Without per-device memory awareness, these methods typically ensure feasibility post hoc by sharding parameters and activations across many devices, increasing synchronization, inflating communication, and underutilizing compute-limiting scalability and efficiency on real datacenter networks. We present NEST, a network-, compute-, and memory-aware device placement framework that unifies model parallelism, topology modeling, and memory feasibility via structured dynamic programming. NEST's DP operates on operator graphs with tensor and expert parallel configurations, explicit allreduce latencies across hierarchical or arbitrary networks, and memory/compute profiles. By factoring parallelism across tensor, pipeline, data, and expert dimensions, NEST defines a principled search space for hybrid strategies while jointly optimizing co-location, network latency, and memory feasibility. Evaluations across diverse hardware and networks show NEST achieves up to 2.43 times higher throughput, better memory efficiency, and improved scalability over state-of-the-art baselines, providing a foundation for co-designing parallelization strategies and datacenter interconnects for next-generation AI infrastructure. The source code of NEST is available at: https://github.com/scai-tech/Nest

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 43 references that are in the map. First posted on arXiv 6 Mar 2026.

Builds on

20 of 43 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  5. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  6. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  7. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  8. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  9. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 134
  10. 2017Device Placement Optimization with Reinforcement Learningcited by 86
  11. 2023ASTRA-sim2.0: Modeling Hierarchical Networks and Disaggregated Systems for Large-model Training at Scalecited by 47
  12. 2021Torch.fx: Practical Program Capture and Transformation for Deep Learning in Pythoncited by 40
8 more
  1. 2024Context Parallelism for Scalable Million-Token Inferencecited by 26
  2. 2020Efficient Algorithms for Device Placement of DNN Graph Operatorscited by 24
  3. 2021Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Modelscited by 16
  4. 2025Mist: Efficient Distributed Training of Large Language Models via Memory-Parallelism Co-Optimizationcited by 7
  5. 2025CATransformers: Carbon Aware Transformers Through Joint Model-Hardware Optimizationcited by 7
  6. 2025Characterizing the Efficiency of Distributed Training: A Power, Performance, and Thermal Perspectivecited by 5
  7. 2024Integrated Hardware Architecture and Device Placement Searchcited by 4
  8. 2025Characterizing Compute-Communication Overlap in GPU-Accelerated Distributed Deep Learning: Performance and Power Implicationscited by 3

Built on by

Not cited yet.