# Decentralized Low-Latency Collaborative Inference via Ensembles on the Edge

arXiv: https://arxiv.org/abs/2206.03165 · HTML: https://mlsanity.com/paper/2206.03165/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 55 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 32)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Channel Pruning for Accelerating Very Deep Neural Networks](https://mlsanity.com/paper/1707.06168/) (2017)
- [End-to-end Optimized Image Compression](https://mlsanity.com/paper/1611.01704/) (2016)
- [Deep Ensembles: A Loss Landscape Perspective](https://mlsanity.com/paper/1912.02757/) (2019)
- [A Survey of Model Compression and Acceleration for Deep Neural Networks](https://mlsanity.com/paper/1710.09282/) (2017)
- [Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing](https://mlsanity.com/paper/1905.10083/) (2019)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Model-Based Deep Learning](https://mlsanity.com/paper/2012.08405/) (2020)
- [Conditional Probability Models for Deep Image Compression](https://mlsanity.com/paper/1801.04260/) (2018)
- [Deep Learning in Mobile and Wireless Networking: A Survey](https://mlsanity.com/paper/1803.04311/) (2018)
- [UVeQFed: Universal Vector Quantization for Federated Learning](https://mlsanity.com/paper/2006.03262/) (2020)
- [Robust Federated Learning in a Heterogeneous Environment](https://mlsanity.com/paper/1906.06629/) (2019)
- [Image and Video Compression with Neural Networks: A Review](https://mlsanity.com/paper/1904.03567/) (2019)
- [JointDNN: An Efficient Training and Inference Engine for Intelligent Mobile Cloud Computing Services](https://mlsanity.com/paper/1801.08618/) (2018)
- [Towards Image Understanding from Deep Compression without Decoding](https://mlsanity.com/paper/1803.06131/) (2018)
- [Federated Learning: A Signal Processing Perspective](https://mlsanity.com/paper/2103.17150/) (2021)
- [Distributed Deep Convolutional Compression for Massive MIMO CSI Feedback](https://mlsanity.com/paper/2003.04684/) (2020)
- [Collaborative Intelligence: Challenges and Opportunities](https://mlsanity.com/paper/2102.06841/) (2021)
- [Comparison and Benchmarking of AI Models and Frameworks on Mobile Devices](https://mlsanity.com/paper/2005.05085/) (2020)
- [Deep Task-Based Quantization](https://mlsanity.com/paper/1908.06845/) (2019)
- [Diversity regularization in deep ensembles](https://mlsanity.com/paper/1802.07881/) (2018)
- [Data-Driven Symbol Detection via Model-Based Machine Learning](https://mlsanity.com/paper/2002.07806/) (2020)
- [Learned Factor Graphs for Inference from Stationary Time Sequences](https://mlsanity.com/paper/2006.03258/) (2020)
- [Group Ensemble: Learning an Ensemble of ConvNets in a single ConvNet](https://mlsanity.com/paper/2007.00649/) (2020)
- [Lightweight Compression of Intermediate Neural Network Features for Collaborative Intelligence](https://mlsanity.com/paper/2105.07102/) (2021)

## Built on by (most cited first, 10 of 10)

- [Position: Collaborative Agentic AI Needs Interoperability Across Ecosystems](https://mlsanity.com/paper/2505.21550/) (2025)
- [Remote Inference over Dynamic Links via Adaptive Rate Deep Task-Oriented Vector Quantization](https://mlsanity.com/paper/2501.02521/) (2025)
- [Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRL](https://mlsanity.com/paper/2512.23310/) (2025)
- [Limited Communications Distributed Optimization via Deep Unfolded Distributed ADMM](https://mlsanity.com/paper/2309.14353/) (2023)
- [Edge-device Collaborative Computing for Multi-view Classification](https://mlsanity.com/paper/2409.15973/) (2024)
- [Robust Federated Inference](https://mlsanity.com/paper/2510.00310/) (2025)
- [Federated Learning for Collaborative Inference Systems: The Case of Early Exit Networks](https://mlsanity.com/paper/2405.04249/) (2024)
- [Multi-Rate Task-Oriented Communication for Multi-Edge Cooperative Inference](https://mlsanity.com/paper/2510.19360/) (2025)
- [SpecFed: Accelerating Federated LLM Inference with Speculative Decoding and Compressed Transmission](https://mlsanity.com/paper/2604.25777/) (2026)
- [Enabling Federated Inference via Unsupervised Consensus Embedding](https://mlsanity.com/paper/2605.05718/) (2026)

## Abstract (arXiv)

The success of deep neural networks (DNNs) is heavily dependent on computational resources. While DNNs are often employed on cloud servers, there is a growing need to operate DNNs on edge devices. Edge devices are typically limited in their computational resources, yet, often multiple edge devices are deployed in the same environment and can reliably communicate with each other. In this work we propose to facilitate the application of DNNs on the edge by allowing multiple users to collaborate during inference to improve their accuracy. Our mechanism, coined em edge ensembles, is based on having diverse predictors at each device, which form an ensemble of models during inference. To mitigate the communication overhead, the users share quantized features, and we propose a method for aggregating multiple decisions into a single inference rule. We analyze the latency induced by edge ensembles, showing that its performance improvement comes at the cost of a minor additional delay under common assumptions on the communication network. Our experiments demonstrate that collaborative inference via edge ensembles equipped with compact DNNs substantially improves the accuracy over having each user infer locally, and can outperform using a single centralized DNN larger than all the networks in the ensemble together.