# Mind Mappings: Enabling Efficient Algorithm-Accelerator Mapping Space Search

arXiv: https://arxiv.org/abs/2103.01489 · HTML: https://mlsanity.com/paper/2103.01489/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 113 references that are in the map.
- First posted on arXiv 2 Mar 2021.

## Builds on (most cited first, 30 of 32)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Going Deeper with Convolutions](https://mlsanity.com/paper/1409.4842/) (2014)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018)
- [Neural Architecture Search with Reinforcement Learning](https://mlsanity.com/paper/1611.01578/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [An Alternative View: When Does SGD Escape Local Minima?](https://mlsanity.com/paper/1802.06175/) (2018)
- [Backpropagation through the Void: Optimizing control variates for black-box gradient estimation](https://mlsanity.com/paper/1711.00123/) (2017)
- [Tensor Comprehensions: Framework-Agnostic High-Performance Machine Learning Abstractions](https://mlsanity.com/paper/1802.04730/) (2018)
- [REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models](https://mlsanity.com/paper/1703.07370/) (2017)
- [Fast low-rank estimation by projected gradient descent: General statistical and algorithmic guarantees](https://mlsanity.com/paper/1509.03025/) (2015)
- [Tiramisu: A Polyhedral Compiler for Expressing Fast and Portable Code](https://mlsanity.com/paper/1804.10694/) (2018)
- [Ithemal: Accurate, Portable and Fast Basic Block Throughput Estimation using Deep Neural Networks](https://mlsanity.com/paper/1808.07412/) (2018)
- [SCALE-Sim: Systolic CNN Accelerator Simulator](https://mlsanity.com/paper/1811.02883/) (2018)
- [Adversarial Variational Optimization of Non-Differentiable Simulators](https://mlsanity.com/paper/1707.07113/) (2017)
- [UCNN: Exploiting Computational Reuse in Deep Neural Networks via Weight Repetition](https://mlsanity.com/paper/1804.06508/) (2018)
- [DiffTune: Optimizing CPU Simulator Parameters with Learned Differentiable Surrogates](https://mlsanity.com/paper/2010.04017/) (2020)
- [GANAX: A Unified MIMD-SIMD Acceleration for Generative Adversarial Networks](https://mlsanity.com/paper/1806.01107/) (2018)
- [Morph: Flexible Acceleration for 3D CNN-based Video Understanding](https://mlsanity.com/paper/1810.06807/) (2018)
- [Reinforcement Learning and Adaptive Sampling for Optimized DNN Compilation](https://mlsanity.com/paper/1905.12799/) (2019)

## Built on by (most cited first, 24 of 24)

- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks](https://mlsanity.com/paper/2107.06419/) (2021)
- [Data-Driven Offline Optimization For Architecting Hardware Accelerators](https://mlsanity.com/paper/2110.11346/) (2021)
- [CoSA: Scheduling by Constrained Optimization for Spatial Accelerators](https://mlsanity.com/paper/2105.01898/) (2021)
- [A Full-Stack Search Technique for Domain Optimized Deep Learning Accelerators](https://mlsanity.com/paper/2105.12842/) (2021)
- [A Hardware Evaluation Framework for Large Language Model Inference](https://mlsanity.com/paper/2312.03134/) (2023)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity](https://mlsanity.com/paper/2305.12718/) (2023)
- [GRANITE: A Graph Neural Network Model for Basic Block Throughput Estimation](https://mlsanity.com/paper/2210.03894/) (2022)
- [Performance Modeling and Workload Analysis of Distributed Large Language Model Training and Inference](https://mlsanity.com/paper/2407.14645/) (2024)
- [ArchGym: An Open-Source Gymnasium for Machine Learning Assisted Architecture Design](https://mlsanity.com/paper/2306.08888/) (2023)
- [SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators](https://mlsanity.com/paper/2405.00790/) (2024)
- [Union: A Unified HW-SW Co-Design Ecosystem in MLIR for Evaluating Tensor Operations on Spatial Accelerators](https://mlsanity.com/paper/2109.07419/) (2021)
- [Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips](https://mlsanity.com/paper/2308.05869/) (2023)
- [DOSA: Differentiable Model-Based One-Loop Search for DNN Accelerators](https://mlsanity.com/paper/2509.10702/) (2025)
- [HW-Aware Initialization of DNN Auto-Tuning to Improve Exploration Time and Robustness](https://mlsanity.com/paper/2205.15568/) (2022)
- [DNNFuser: Generative Pre-Trained Transformer as a Generalized Mapper for Layer Fusion in DNN Accelerators](https://mlsanity.com/paper/2201.11218/) (2022)
- [Demystifying Map Space Exploration for NPUs](https://mlsanity.com/paper/2210.03731/) (2022)
- [SALSA: Simulated Annealing based Loop-Ordering Scheduler for DNN Accelerators](https://mlsanity.com/paper/2304.12931/) (2023)
- [Target-independent XLA optimization using Reinforcement Learning](https://mlsanity.com/paper/2308.14364/) (2023)
- [Obsidian: Cooperative State-Space Exploration for Performant Inference on Secure ML Accelerators](https://mlsanity.com/paper/2409.02817/) (2024)
- [AIRCHITECT v2: Learning the Hardware Accelerator Design Space through Unified Representations](https://mlsanity.com/paper/2501.09954/) (2025)
- [KAPLA: Pragmatic Representation and Fast Solving of Scalable NN Accelerator Dataflow](https://mlsanity.com/paper/2306.15676/) (2023)
- [Multi-Agent Reinforcement Learning for Sample-Efficient Deep Neural Network Mapping](https://mlsanity.com/paper/2507.16249/) (2025)

## Abstract (arXiv)

Modern day computing increasingly relies on specialization to satiate growing performance and efficiency requirements. A core challenge in designing such specialized hardware architectures is how to perform mapping space search, i.e., search for an optimal mapping from algorithm to hardware. Prior work shows that choosing an inefficient mapping can lead to multiplicative-factor efficiency overheads. Additionally, the search space is not only large but also non-convex and non-smooth, precluding advanced search techniques. As a result, previous works are forced to implement mapping space search using expert choices or sub-optimal search heuristics. This work proposes Mind Mappings, a novel gradient-based search method for algorithm-accelerator mapping space search. The key idea is to derive a smooth, differentiable approximation to the otherwise non-smooth, non-convex search space. With a smooth, differentiable approximation, we can leverage efficient gradient-based search algorithms to find high-quality mappings. We extensively compare Mind Mappings to black-box optimization schemes used in prior work. When tasked to find mappings for two important workloads (CNN and MTTKRP), the proposed search finds mappings that achieve an average 1.40×, 1.76×, and 1.29× (when run for a fixed number of steps) and 3.16×, 4.19×, and 2.90× (when run for a fixed amount of time) better energy-delay product (EDP) relative to Simulated Annealing, Genetic Algorithms and Reinforcement Learning, respectively. Meanwhile, Mind Mappings returns mappings with only 5.32× higher EDP than a possibly unachievable theoretical lower-bound, indicating proximity to the global optima.