# Supervised Compression for Resource-Constrained Edge Computing Systems

arXiv: https://arxiv.org/abs/2108.11898 · HTML: https://mlsanity.com/paper/2108.11898/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 60 references that are in the map.
- First posted on arXiv 21 Aug 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs](https://mlsanity.com/paper/1606.00915/) (2016)
- [Rethinking Atrous Convolution for Semantic Image Segmentation](https://mlsanity.com/paper/1706.05587/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Variational image compression with a scale hyperprior](https://mlsanity.com/paper/1802.01436/) (2018)
- [Deep Variational Information Bottleneck](https://mlsanity.com/paper/1612.00410/) (2016)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Joint Autoregressive and Hierarchical Priors for Learned Image Compression](https://mlsanity.com/paper/1809.02736/) (2018)
- [Do Deep Nets Really Need to be Deep?](https://mlsanity.com/paper/1312.6184/) (2013)
- [Rethinking ImageNet Pre-training](https://mlsanity.com/paper/1811.08883/) (2018)
- [Lossy Image Compression with Compressive Autoencoders](https://mlsanity.com/paper/1703.00395/) (2017)
- [CompressAI: a PyTorch library and evaluation platform for end-to-end compression research](https://mlsanity.com/paper/2011.03029/) (2020)
- [Channel-wise Autoregressive Entropy Models for Learned Image Compression](https://mlsanity.com/paper/2007.08739/) (2020)
- [Full Resolution Image Compression with Recurrent Neural Networks](https://mlsanity.com/paper/1608.05148/) (2016)
- [Density Modeling of Images using a Generalized Normalization Transformation](https://mlsanity.com/paper/1511.06281/) (2015)

## Built on by (most cited first, 30 of 31)

- [Beyond Transmitting Bits: Context, Semantics, and Task-Oriented Communications](https://mlsanity.com/paper/2207.09353/) (2022)
- [Improving Statistical Fidelity for Neural Image Compression with Implicit Local Likelihood Models](https://mlsanity.com/paper/2301.11189/) (2023)
- [An Introduction to Neural Data Compression](https://mlsanity.com/paper/2202.06533/) (2022)
- [BottleFit: Learning Compressed Representations in Deep Neural Networks for Effective and Efficient Split Computing](https://mlsanity.com/paper/2201.02693/) (2022)
- [SC2 Benchmark: Supervised Compression for Split Computing](https://mlsanity.com/paper/2203.08875/) (2022)
- [Generative Visual Compression: A Review](https://mlsanity.com/paper/2402.02140/) (2024)
- [FrankenSplit: Efficient Neural Feature Compression with Shallow Variational Bottleneck Injection for Mobile Edge Computing](https://mlsanity.com/paper/2302.10681/) (2023)
- [Lossy Neural Compression for Geospatial Analytics: A Review](https://mlsanity.com/paper/2503.01505/) (2025)
- [C3-SL: Circular Convolution-Based Batch-Wise Compression for Communication-Efficient Split Learning](https://mlsanity.com/paper/2207.12397/) (2022)
- [Feature Compression for Rate Constrained Object Detection on the Edge](https://mlsanity.com/paper/2204.07314/) (2022)
- [I-SPLIT: Deep Network Interpretability for Split Computing](https://mlsanity.com/paper/2209.11607/) (2022)
- [FOOL: Addressing the Downlink Bottleneck in Satellite Computing with Neural Feature Compression](https://mlsanity.com/paper/2403.16677/) (2024)
- [A Multi-task Supervised Compression Model for Split Computing](https://mlsanity.com/paper/2501.01420/) (2025)
- [Distributed Semantic Segmentation with Efficient Joint Source and Task Decoding](https://mlsanity.com/paper/2407.11224/) (2024)
- [Slimmable Encoders for Flexible Split DNNs in Bandwidth and Resource Constrained IoT Systems](https://mlsanity.com/paper/2306.12691/) (2023)
- [Learned Point Cloud Compression for Classification](https://mlsanity.com/paper/2308.05959/) (2023)
- [Progressive Neural Compression for Adaptive Image Offloading under Timing Constraints](https://mlsanity.com/paper/2310.05306/) (2023)
- [On the Impact of Black-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2403.17154/) (2024)
- [Adversarial Attacks to Latent Representations of Distributed Neural Networks in Split Computing](https://mlsanity.com/paper/2309.17401/) (2023)
- [Communication Efficient Split Learning of ViTs with Attention-based Double Compression](https://mlsanity.com/paper/2509.15058/) (2025)
- [Model-Aware Rate-Distortion Limits for Task-Oriented Source Coding](https://mlsanity.com/paper/2602.12866/) (2026)
- [Resilience of Entropy Model in Distributed Neural Networks](https://mlsanity.com/paper/2403.00942/) (2024)
- [On the Impact of White-box Deployment Strategies for Edge AI on Latency and Model Performance](https://mlsanity.com/paper/2411.00907/) (2024)
- [Adversarial Robustness of Bottleneck Injected Deep Neural Networks for Task-Oriented Communication](https://mlsanity.com/paper/2412.10265/) (2024)
- [Why Should the Server Do It All?: A Scalable, Versatile, and Model-Agnostic Framework for Server-Light DNN Inference over Massively Distributed Clients via Training-Free Intermediate Feature Compression](https://mlsanity.com/paper/2511.11608/) (2025)
- [Fault-Tolerant Collaborative Inference through the Edge-PRUNE Framework](https://mlsanity.com/paper/2206.08152/) (2022)
- [Codebook-Based Adaptive Feature Compression With Semantic Enhancement for Edge-Cloud Systems](https://mlsanity.com/paper/2509.18481/) (2025)
- [Range Asymmetric Numeral Systems-Based Lightweight Intermediate Feature Compression for Split Computing of Deep Neural Networks](https://mlsanity.com/paper/2511.11664/) (2025)
- [Trainable Bitwise Soft Quantization for Input Feature Compression](https://mlsanity.com/paper/2603.05172/) (2026)
- [Adaptive Transform Coding for Semantic Compression](https://mlsanity.com/paper/2604.26492/) (2026)

## Abstract (arXiv)

There has been much interest in deploying deep learning algorithms on low-powered devices, including smartphones, drones, and medical sensors. However, full-scale deep neural networks are often too resource-intensive in terms of energy and storage. As a result, the bulk part of the machine learning operation is therefore often carried out on an edge server, where the data is compressed and transmitted. However, compressing data (such as images) leads to transmitting information irrelevant to the supervised task. Another popular approach is to split the deep network between the device and the server while compressing intermediate features. To date, however, such split computing strategies have barely outperformed the aforementioned naive data compression baselines due to their inefficient approaches to feature compression. This paper adopts ideas from knowledge distillation and neural image compression to compress intermediate feature representations more efficiently. Our supervised compression approach uses a teacher model and a student model with a stochastic bottleneck and learnable prior for entropy coding (Entropic Student). We compare our approach to various neural image and feature compression baselines in three vision tasks and found that it achieves better supervised rate-distortion performance while maintaining smaller end-to-end latency. We furthermore show that the learned feature representations can be tuned to serve multiple downstream tasks.