# Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training

arXiv: https://arxiv.org/abs/2006.03318 · HTML: https://mlsanity.com/paper/2006.03318/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jun 2020.

## Built on by (most cited first, 22 of 22)

- [Vidur: A Large-Scale Simulation Framework For LLM Inference](https://mlsanity.com/paper/2405.05465/) (2024)
- [A Runtime-Based Computational Performance Predictor for Deep Neural Network Training](https://mlsanity.com/paper/2102.00527/) (2021)
- [Aryl: An Elastic Cluster Scheduler for Deep Learning](https://mlsanity.com/paper/2202.07896/) (2022)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [vTrain: A Simulation Framework for Evaluating Cost-effective and Compute-optimal Large Language Model Training](https://mlsanity.com/paper/2312.12391/) (2023)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training](https://mlsanity.com/paper/2205.02473/) (2022)
- [The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deployment](https://mlsanity.com/paper/2302.06117/) (2023)
- [Building a Performance Model for Deep Learning Recommendation Model Training on GPUs](https://mlsanity.com/paper/2201.07821/) (2022)
- [Characterizing and Optimizing LLM Inference Workloads on CPU-GPU Coupled Architectures](https://mlsanity.com/paper/2504.11750/) (2025)
- [Horizontally Fused Training Array: An Effective Hardware Utilization Squeezer for Training Novel Deep Learning Models](https://mlsanity.com/paper/2102.02344/) (2021)
- [Lumos: Efficient Performance Modeling and Estimation for Large-scale LLM Training](https://mlsanity.com/paper/2504.09307/) (2025)
- [Mystique: Enabling Accurate and Scalable Generation of Production AI Benchmarks](https://mlsanity.com/paper/2301.04122/) (2022)
- [Efficient Fine-Grained GPU Performance Modeling for Distributed Deep Learning of LLM](https://mlsanity.com/paper/2509.22832/) (2025)
- [The Grand Illusion: The Myth of Software Portability and Implications for ML Progress](https://mlsanity.com/paper/2309.07181/) (2023)
- [AscendOptimizer: Episodic Agent for Ascend NPU Operator Optimization](https://mlsanity.com/paper/2603.23566/) (2026)
- [DistIR: An Intermediate Representation and Simulator for Efficient Neural Network Distribution](https://mlsanity.com/paper/2111.05426/) (2021)
- [Echo: Simulating Distributed Training At Scale](https://mlsanity.com/paper/2412.12487/) (2024)
- [Maya: Optimizing Deep Learning Training Workloads using GPU Runtime Emulation](https://mlsanity.com/paper/2503.20191/) (2025)
- [Arena: Efficiently Training Large Models via Dynamic Scheduling and Adaptive Parallelism Co-Design](https://mlsanity.com/paper/2403.16125/) (2024)
- [Phantora: Maximizing Code Reuse in Simulation-based Machine Learning System Performance Estimation](https://mlsanity.com/paper/2505.01616/) (2025)
- [A Few GPUs, A Whole Lotta Scale: Faithful LLM Training Emulation with PrismLLM](https://mlsanity.com/paper/2605.15617/) (2026)

## Abstract (arXiv)

Modern deep neural network (DNN) training jobs use complex and heterogeneous software/hardware stacks. The efficacy of software-level optimizations can vary significantly when used in different deployment configurations. It is onerous and error-prone for ML practitioners and system developers to implement each optimization separately, and determine which ones will improve performance in their own configurations. Unfortunately, existing profiling tools do not aim to answer predictive questions such as "How will optimization X affect the performance of my model?". We address this critical limitation, and proposes a new profiling tool, Daydream, to help programmers efficiently explore the efficacy of DNN optimizations. Daydream models DNN execution with a fine-grained dependency graph based on low-level traces collected by CUPTI, and predicts runtime by simulating execution based on the dependency graph. Daydream maps the low-level traces using DNN domain-specific knowledge, and introduces a set of graph-transformation primitives that can easily model a wide variety of optimizations. We show that Daydream is able to model most mainstream DNN optimization techniques, and accurately predict the efficacy of optimizations that will result in significant performance improvements.