# Polymorph: Energy-Efficient Multi-Label Classification for Video Streams on Embedded Devices

arXiv: https://arxiv.org/abs/2507.14959 · HTML: https://mlsanity.com/paper/2507.14959/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 50 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Builds on (most cited first, 30 of 32)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [AdaViT: Adaptive Tokens for Efficient Vision Transformer](https://mlsanity.com/paper/2112.07658/) (2021)
- [Multi-Label Image Recognition with Graph Convolutional Networks](https://mlsanity.com/paper/1904.03582/) (2019)
- [A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions](https://mlsanity.com/paper/2006.02903/) (2020)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Adaptive Token Sampling For Efficient Vision Transformers](https://mlsanity.com/paper/2111.15667/) (2021)
- [Mixture of LoRA Experts](https://mlsanity.com/paper/2404.13628/) (2024)
- [S-LoRA: Serving Thousands of Concurrent LoRA Adapters](https://mlsanity.com/paper/2311.03285/) (2023)
- [TAO: A Large-Scale Benchmark for Tracking Any Object](https://mlsanity.com/paper/2005.10356/) (2020)
- [Learned Token Pruning for Transformers](https://mlsanity.com/paper/2107.00910/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Learning to Route Among Specialized Experts for Zero-Shot Generalization](https://mlsanity.com/paper/2402.05859/) (2024)
- [HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformers](https://mlsanity.com/paper/2211.08110/) (2022)
- [Transformer-based Dual Relation Graph for Multi-label Image Recognition](https://mlsanity.com/paper/2110.04722/) (2021)
- [Agile-Quant: Activation-Guided Quantization for Faster Inference of LLMs on the Edge](https://mlsanity.com/paper/2312.05693/) (2023)
- [Real-Time Video Inference on Edge Devices via Adaptive Model Streaming](https://mlsanity.com/paper/2006.06628/) (2020)
- [SPMTrack: Spatio-Temporal Parameter-Efficient Fine-Tuning with Mixture of Experts for Scalable Visual Tracking](https://mlsanity.com/paper/2503.18338/) (2025)
- [Edge-MoE: Memory-Efficient Multi-Task Vision Transformer Architecture with Task-level Sparsity via Mixture-of-Experts](https://mlsanity.com/paper/2305.18691/) (2023)
- [A Smart, Efficient, and Reliable Parking Surveillance System with Edge Artificial Intelligence on IoT Devices](https://mlsanity.com/paper/2001.00269/) (2020)
- [Mobile V-MoEs: Scaling Down Vision Transformers via Sparse Mixture-of-Experts](https://mlsanity.com/paper/2309.04354/) (2023)
- [Uni-AdaFocus: Spatial-temporal Dynamic Computation for Video Recognition](https://mlsanity.com/paper/2412.11228/) (2024)
- [Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception](https://mlsanity.com/paper/2106.05665/) (2021)
- [Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems](https://mlsanity.com/paper/2304.10892/) (2023)
- [TOD: Transprecise Object Detection to Maximise Real-Time Accuracy on the Edge](https://mlsanity.com/paper/2105.08668/) (2021)
- [Tiny Models are the Computational Saver for Large Models](https://mlsanity.com/paper/2403.17726/) (2024)

## Abstract (arXiv)

Real-time multi-label video classification on embedded devices is constrained by limited compute and energy budgets. Yet, video streams exhibit structural properties such as label sparsity, temporal continuity, and label co-occurrence that can be leveraged for more efficient inference. We introduce Polymorph, a context-aware framework that activates a minimal set of lightweight Low Rank Adapters (LoRA) per frame. Each adapter specializes in a subset of classes derived from co-occurrence patterns and is implemented as a LoRA weight over a shared backbone. At runtime, Polymorph dynamically selects and composes only the adapters needed to cover the active labels, avoiding full-model switching and weight merging. This modular strategy improves scalability while reducing latency and energy overhead. Polymorph achieves 40% lower energy consumption and improves mAP by 9 points over strong baselines on the TAO dataset. Polymorph is open source at https://github.com/inference-serving/polymorph/.