# H2H: Heterogeneous Model to Heterogeneous System Mapping with Computation and Communication Awareness

arXiv: https://arxiv.org/abs/2204.13852 · HTML: https://mlsanity.com/paper/2204.13852/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 35 references that are in the map.
- First posted on arXiv 29 Apr 2022.

## Builds on (most cited first, 9 of 9)

- [Deep Auxiliary Learning for Visual Localization and Odometry](https://mlsanity.com/paper/1803.03642/) (2018)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA](https://mlsanity.com/paper/1612.00694/) (2016)
- [A Dataset and Benchmark for Large-scale Multi-modal Face Anti-spoofing](https://mlsanity.com/paper/1812.00408/) (2018)
- [Achieving Super-Linear Speedup across Multi-FPGA for Real-Time DNN Inference](https://mlsanity.com/paper/1907.08985/) (2019)
- [Concurrent Activity Recognition with Multimodal CNN-LSTM Structure](https://mlsanity.com/paper/1702.01638/) (2017)
- [A Multimodal Recommender System for Large-scale Assortment Generation in E-commerce](https://mlsanity.com/paper/1806.11226/) (2018)
- [Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems](https://mlsanity.com/paper/2104.04000/) (2021)
- [Compiling Deep Learning Models for Custom Hardware Accelerators](https://mlsanity.com/paper/1708.00117/) (2017)

## Built on by (most cited first, 6 of 6)

- [Harmonic-NAS: Hardware-Aware Multimodal Neural Architecture Search on Resource-constrained Devices](https://mlsanity.com/paper/2309.06612/) (2023)
- [SCAR: Scheduling Multi-Model AI Workloads on Heterogeneous Multi-Chiplet Module Accelerators](https://mlsanity.com/paper/2405.00790/) (2024)
- [Shared Memory-contention-aware Concurrent DNN Execution for Diversely Heterogeneous System-on-Chips](https://mlsanity.com/paper/2308.05869/) (2023)
- [Ev-Edge: Efficient Execution of Event-based Vision Algorithms on Commodity Edge Platforms](https://mlsanity.com/paper/2403.15717/) (2024)
- [MARS: Exploiting Multi-Level Parallelism for DNN Workloads on Adaptive Multi-Accelerator Systems](https://mlsanity.com/paper/2307.12234/) (2023)
- [Performance Implications of Multi-Chiplet Neural Processing Units on Autonomous Driving Perception](https://mlsanity.com/paper/2411.16007/) (2024)

## Abstract (arXiv)

The complex nature of real-world problems calls for heterogeneity in both machine learning (ML) models and hardware systems. The heterogeneity in ML models comes from multi-sensor perceiving and multi-task learning, i.e., multi-modality multi-task (MMMT), resulting in diverse deep neural network (DNN) layers and computation patterns. The heterogeneity in systems comes from diverse processing components, as it becomes the prevailing method to integrate multiple dedicated accelerators into one system. Therefore, a new problem emerges: heterogeneous model to heterogeneous system mapping (H2H). While previous mapping algorithms mostly focus on efficient computations, in this work, we argue that it is indispensable to consider computation and communication simultaneously for better system efficiency. We propose a novel H2H mapping algorithm with both computation and communication awareness; by slightly trading computation for communication, the system overall latency and energy consumption can be largely reduced. The superior performance of our work is evaluated based on MAESTRO modeling, demonstrating 15%-74% latency reduction and 23%-64% energy reduction compared with existing computation-prioritized mapping algorithms.