# Ordering Chaos: Memory-Aware Scheduling of Irregularly Wired Neural Networks for Edge Devices

arXiv: https://arxiv.org/abs/2003.02369 · HTML: https://mlsanity.com/paper/2003.02369/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 Mar 2020.

## Built on by (most cited first, 27 of 27)

- [On-Device Training Under 256KB Memory](https://mlsanity.com/paper/2206.15472/) (2022)
- [MCUNetV2: Memory-Efficient Patch-based Inference for Tiny Deep Learning](https://mlsanity.com/paper/2110.15352/) (2021)
- [Tiny Machine Learning: Progress and Futures](https://mlsanity.com/paper/2403.19076/) (2024)
- [PockEngine: Sparse and Efficient Fine-tuning in a Pocket](https://mlsanity.com/paper/2310.17752/) (2023)
- [Neural Topological Ordering for Computation Graphs](https://mlsanity.com/paper/2207.05899/) (2022)
- [Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity](https://mlsanity.com/paper/2309.06626/) (2023)
- [Dynamic ConvNets on Tiny Devices via Nested Sparsity](https://mlsanity.com/paper/2203.03324/) (2022)
- [RESPECT: Reinforcement Learning based Edge Scheduling on Pipelined Coral Edge TPUs](https://mlsanity.com/paper/2304.04716/) (2023)
- [OLLA: Optimizing the Lifetime and Location of Arrays to Reduce the Memory Usage of Neural Networks](https://mlsanity.com/paper/2210.12924/) (2022)
- [Differentiable Combinatorial Scheduling at Scale](https://mlsanity.com/paper/2406.06593/) (2024)
- [Accelerating Exact Combinatorial Optimization via RL-based Initialization -- A Case Study in Scheduling](https://mlsanity.com/paper/2308.11652/) (2023)
- [SoD2: Statically Optimizing Dynamic Deep Neural Network](https://mlsanity.com/paper/2403.00176/) (2024)
- [How to Reach Real-Time AI on Consumer Devices? Solutions for Programmable and Custom Architectures](https://mlsanity.com/paper/2106.15021/) (2021)
- [Baechi: Fast Device Placement of Machine Learning Graphs](https://mlsanity.com/paper/2301.08695/) (2023)
- [Fused Depthwise Tiling for Memory Optimization in TinyML Deep Neural Network Inference](https://mlsanity.com/paper/2303.17878/) (2023)
- [Memory-aware Scheduling for Complex Wired Networks with Iterative Graph Optimization](https://mlsanity.com/paper/2308.13898/) (2023)
- [Terra: Imperative-Symbolic Co-Execution of Imperative Deep Learning Programs](https://mlsanity.com/paper/2201.09210/) (2022)
- [AGO: Boosting Mobile AI Inference Performance by Removing Constraints on Graph Optimization](https://mlsanity.com/paper/2212.01005/) (2022)
- [YFlows: Systematic Dataflow Exploration and Code Generation for Efficient Neural Network Inference using SIMD Architectures on CPUs](https://mlsanity.com/paper/2310.00574/) (2023)
- [vMCU: Coordinated Memory Management and Kernel Optimization for DNN Inference on MCUs](https://mlsanity.com/paper/2406.06542/) (2024)
- [DiviML: A Module-based Heuristic for Mapping Neural Networks onto Heterogeneous Platforms](https://mlsanity.com/paper/2308.00127/) (2023)
- [ROAM: memory-efficient large DNN training via optimized operator ordering and memory layout](https://mlsanity.com/paper/2310.19295/) (2023)
- [Practical Performance Guarantees for Pipelined DNN Inference](https://mlsanity.com/paper/2311.03703/) (2023)
- [Combined Scheduling, Memory Allocation and Tensor Replacement for Minimizing Off-Chip Data Accesses of DNN Accelerators](https://mlsanity.com/paper/2311.18246/) (2023)
- [Data Overfitting for On-Device Super-Resolution with Dynamic Algorithm and Compiler Co-Design](https://mlsanity.com/paper/2407.02813/) (2024)
- [Differentiable Initialization-Accelerated CPU-GPU Hybrid Combinatorial Scheduling](https://mlsanity.com/paper/2603.28943/) (2026)
- [Empowering Hybrid Attention Models on NPUs](https://mlsanity.com/paper/2609.32114/) (2026)

## Abstract (arXiv)

Recent advances demonstrate that irregularly wired neural networks from Neural Architecture Search (NAS) and Random Wiring can not only automate the design of deep neural networks but also emit models that outperform previous manual designs. These designs are especially effective while designing neural architectures under hard resource constraints (memory, MACs, . . . ) which highlights the importance of this class of designing neural networks. However, such a move creates complication in the previously streamlined pattern of execution. In fact one of the main challenges is that the order of such nodes in the neural network significantly effects the memory footprint of the intermediate activations. Current compilers do not schedule with regard to activation memory footprint that it significantly increases its peak compared to the optimum, rendering it not applicable for edge devices. To address this standing issue, we present a memory-aware compiler, dubbed SERENITY, that utilizes dynamic programming to find a sequence that finds a schedule with optimal memory footprint. Our solution also comprises of graph rewriting technique that allows further reduction beyond the optimum. As such, SERENITY achieves optimal peak memory, and the graph rewriting technique further improves this resulting in 1.68x improvement with dynamic programming-based scheduler and 1.86x with graph rewriting, against TensorFlow Lite with less than one minute overhead.