mlsanityData to 14 Oct 2026

Tessel: Boosting Distributed Execution of Large DNN Models via Flexible Schedule Search

arXiv 2311.15269 ↗First posted 26 Nov 2023cs.DC cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Increasingly complex and diverse deep neural network (DNN) models necessitate distributing the execution across multiple devices for training and inference tasks, and also require carefully planned schedules for performance. However, existing practices often rely on predefined schedules that may not fully exploit the benefits of emerging diverse model-aware operator placement strategies. Handcrafting high-efficiency schedules can be challenging due to the large and varying schedule space. This paper presents Tessel, an automated system that searches for efficient schedules for distributed DNN training and inference for diverse operator placement strategies. To reduce search costs, Tessel leverages the insight that the most efficient schedules often exhibit repetitive pattern (repetend) across different data inputs. This leads to a two-phase approach: repetend construction and schedule completion. By exploring schedules for various operator placement strategies, Tessel significantly improves both training and inference performance. Experiments with representative DNN models demonstrate that Tessel achieves up to 5.5x training performance speedup and up to 38% inference latency reduction.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 23 of its 46 references that are in the map. First posted on arXiv 26 Nov 2023.

Builds on

23 of 46 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  4. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  5. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  6. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  10. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  11. 2021ZeRO-Offload: Democratizing Billion-Scale Model Trainingcited by 341
  12. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
11 more
  1. 2018Beyond Data and Model Parallelism for Deep Neural Networkscited by 179
  2. 2021GSPMD: General and Scalable Parallelization for ML Computation Graphscited by 111
  3. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 101
  4. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  5. 2020HetPipe: Enabling Large DNN Training on (Whimpy) Heterogeneous GPU Clusters through Integration of Pipelined Model Parallelism and Data Parallelismcited by 46
  6. 2018Supporting Very Large Models using Automatic Dataflow Graph Partitioningcited by 43
  7. 2021Varuna: Scalable, Low-cost Training of Massive Deep Learning Modelscited by 40
  8. 2021Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-trainingcited by 22
  9. 2022On Optimizing the Communication of Model Parallelismcited by 19
  10. 2022HelixFold: An Efficient Implementation of AlphaFold2 using PaddlePaddlecited by 12
  11. 2023SuperScaler: Supporting Flexible DNN Parallelization via a Unified Abstractioncited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 1.

2 papers.

  1. 2025OctoPipe: Reducing Pipeline Bubbles for Heterogeneous Models via Co-Optimizing Partitioning, Placement, and Schedulingcited by 2
  2. 2024Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Designcited by 0