# CARB: A Characterization-Guided Framework for CNN Inference Cost Prediction and Deployment Screening

arXiv: https://arxiv.org/abs/2608.10506 · HTML: https://mlsanity.com/paper/2608.10506/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 27 references that are in the map.
- First posted on arXiv 11 Aug 2026.

## Builds on (most cited first, 16 of 16)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [ProxylessNAS: Direct Neural Architecture Search on Target Task and Hardware](https://mlsanity.com/paper/1812.00332/) (2018)
- [FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.03443/) (2018)
- [Designing Energy-Efficient Convolutional Neural Networks using Energy-Aware Pruning](https://mlsanity.com/paper/1611.05128/) (2016)
- [An Analysis of Deep Neural Network Models for Practical Applications](https://mlsanity.com/paper/1605.07678/) (2016)
- [NeuralPower: Predict and Deploy Energy-Efficient Convolutional Neural Networks](https://mlsanity.com/paper/1710.05420/) (2017)
- [Forecasting GPU Performance for Deep Learning Training and Inference](https://mlsanity.com/paper/2407.13853/) (2024)
- [HyperPower: Power- and Memory-Constrained Hyper-Parameter Optimization for Neural Networks](https://mlsanity.com/paper/1712.02446/) (2017)
- [Characterizing Deep Learning Training Workloads on Alibaba-PAI](https://mlsanity.com/paper/1910.05930/) (2019)
- [DVFS-Aware DNN Inference on GPUs: Latency Modeling and Performance Analysis](https://mlsanity.com/paper/2502.06295/) (2025)
- [Energy-based Tuning of Convolutional Neural Networks on Multi-GPUs](https://mlsanity.com/paper/1808.00286/) (2018)
- [MAPLE: Microprocessor A Priori for Latency Estimation](https://mlsanity.com/paper/2111.15106/) (2021)
- [Latenrgy: Model Agnostic Latency and Energy Consumption Prediction for Binary Classifiers](https://mlsanity.com/paper/2412.19241/) (2024)

## Abstract (arXiv)

Accurate pre-deployment estimation of CNN inference cost--energy, latency, and peak memory--is increasingly critical as models are deployed on resource-constrained GPU platforms. Existing approaches rely on FLOPs, latency measurements, or single-device profiling as energy proxies, overlooking the non-linear interactions between architectural design and hardware load. We present a workload characterization study of 13 419 CNN configurations on two GPU platforms (RTX 5090 and RTX 3080) under GPU telemetry, revealing that energy, latency, and memory exhibit fundamentally distinct scaling behaviors: energy and latency diverge by 3x under high computational demand, and cross-GPU transferability differs by target--energy and latency require platform-specific models while memory transfers well across the two tested platforms. Building on these characterization findings, we develop CARB, a cascade-blended ensemble that jointly predicts all three targets with R2 ~0.99, and a two-stage deployment screening workflow that eliminates over 90% of candidates in seconds, reducing large design spaces to a Pareto-prioritized shortlist validated against real hardware.