# Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception

arXiv: https://arxiv.org/abs/2106.05665 · HTML: https://mlsanity.com/paper/2106.05665/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 10 Jun 2021.

## Builds on (most cited first, 24 of 24)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Focal Loss for Dense Object Detection](https://mlsanity.com/paper/1708.02002/) (2017)
- [Argoverse: 3D Tracking and Forecasting with Rich Maps](https://mlsanity.com/paper/1911.02620/) (2019)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [Tracking without bells and whistles](https://mlsanity.com/paper/1903.05625/) (2019)
- [Scaled-YOLOv4: Scaling Cross Stage Partial Network](https://mlsanity.com/paper/2011.08036/) (2020)
- [Speed/accuracy trade-offs for modern convolutional object detectors](https://mlsanity.com/paper/1611.10012/) (2016)
- [TIDE: A General Toolbox for Identifying Object Detection Errors](https://mlsanity.com/paper/2008.08115/) (2020)
- [Action Branching Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1711.08946/) (2017)
- [T-CNN: Tubelets with Convolutional Neural Networks for Object Detection from Videos](https://mlsanity.com/paper/1604.02532/) (2016)
- [Ekya: Continuous Learning of Video Analytics Models on Edge Compute Servers](https://mlsanity.com/paper/2012.10557/) (2020)
- [Real-Time Reinforcement Learning](https://mlsanity.com/paper/1911.04448/) (2019)
- [AdaScale: Towards Real-time Video Object Detection Using Adaptive Scaling](https://mlsanity.com/paper/1902.02910/) (2019)
- [ApproxDet: Content and Contention-Aware Approximate Object Detection for Mobiles](https://mlsanity.com/paper/2010.10754/) (2020)
- [Reactive Reinforcement Learning in Asynchronous Environments](https://mlsanity.com/paper/1802.06139/) (2018)
- [SENSEI: Aligning Video Streaming Quality with Dynamic User Sensitivity](https://mlsanity.com/paper/2008.04687/) (2020)

## Built on by (most cited first, 5 of 5)

- [Instance-Warp: Saliency Guided Image Warping for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2403.12712/) (2024)
- [Real-time Stereo-based 3D Object Detection for Streaming Perception](https://mlsanity.com/paper/2410.12394/) (2024)
- [HopTrack: A Real-time Multi-Object Tracking System for Embedded Devices](https://mlsanity.com/paper/2411.00608/) (2024)
- [Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices](https://mlsanity.com/paper/2507.14959/) (2025)
- [4DSegStreamer: Streaming 4D Panoptic Segmentation via Dual Threads](https://mlsanity.com/paper/2510.17664/) (2025)

## Abstract (arXiv)

Real-time perception requires planned resource utilization. Computational planning in real-time perception is governed by two considerations -- accuracy and latency. There exist run-time decisions (e.g. choice of input resolution) that induce tradeoffs affecting performance on a given hardware, arising from intrinsic (content, e.g. scene clutter) and extrinsic (system, e.g. resource contention) characteristics. Earlier runtime execution frameworks employed rule-based decision algorithms and operated with a fixed algorithm latency budget to balance these concerns, which is sub-optimal and inflexible. We propose Chanakya, a learned approximate execution framework that naturally derives from the streaming perception paradigm, to automatically learn decisions induced by these tradeoffs instead. Chanakya is trained via novel rewards balancing accuracy and latency implicitly, without approximating either objectives. Chanakya simultaneously considers intrinsic and extrinsic context, and predicts decisions in a flexible manner. Chanakya, designed with low overhead in mind, outperforms state-of-the-art static and dynamic execution policies on public datasets on both server GPUs and edge devices.