# Collaborative Learning via Prediction Consensus

arXiv: https://arxiv.org/abs/2305.18497 · HTML: https://mlsanity.com/paper/2305.18497/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 49 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Federated Learning Based on Dynamic Regularization](https://mlsanity.com/paper/2111.04263/) (2021)
- [Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1710.05006/) (2017)
- [Data-Free Knowledge Distillation for Heterogeneous Federated Learning](https://mlsanity.com/paper/2105.10056/) (2021)
- [A Unified Theory of Decentralized SGD with Changing Topology and Local Updates](https://mlsanity.com/paper/2003.10422/) (2020)
- [FLTrust: Byzantine-robust Federated Learning via Trust Bootstrapping](https://mlsanity.com/paper/2012.13995/) (2020)
- [BCN20000: Dermoscopic Lesions in the Wild](https://mlsanity.com/paper/1908.02288/) (2019)
- [Skin Lesion Analysis toward Melanoma Detection: A Challenge at the International Symposium on Biomedical Imaging (ISBI) 2016, hosted by the International Skin Imaging Collaboration (ISIC)](https://mlsanity.com/paper/1605.01397/) (2016)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Stochastic Gradient Push for Distributed Deep Learning](https://mlsanity.com/paper/1811.10792/) (2018)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [Byzantine-Robust Learning on Heterogeneous Datasets via Bucketing](https://mlsanity.com/paper/2006.09365/) (2020)
- [FLamby: Datasets and Benchmarks for Cross-Silo Federated Learning in Realistic Healthcare Settings](https://mlsanity.com/paper/2210.04620/) (2022)
- [Personalized and Private Peer-to-Peer Machine Learning](https://mlsanity.com/paper/1705.08435/) (2017)
- [SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Training](https://mlsanity.com/paper/2106.01432/) (2021)
- [Refined Convergence and Topology Learning for Decentralized SGD with Heterogeneous Data](https://mlsanity.com/paper/2204.04452/) (2022)
- [Architecture Agnostic Federated Learning for Neural Networks](https://mlsanity.com/paper/2202.07757/) (2022)
- [Reconstructing Training Data from Model Gradient, Provably](https://mlsanity.com/paper/2212.03714/) (2022)
- [Data-heterogeneity-aware Mixing for Decentralized Learning](https://mlsanity.com/paper/2204.06477/) (2022)
- [Find Your Friends: Personalized Federated Learning with the Right Collaborators](https://mlsanity.com/paper/2210.06597/) (2022)
- [On the Distributed Optimization over Directed Networks](https://mlsanity.com/paper/1510.02146/) (2015)
- [Collective Learning](https://mlsanity.com/paper/1912.02580/) (2019)
- [Test-time Collective Prediction](https://mlsanity.com/paper/2106.12012/) (2021)

## Built on by (most cited first, 8 of 8)

- [Personalized Collaborative Fine-Tuning for On-Device Large Language Models](https://mlsanity.com/paper/2404.09753/) (2024)
- [On-Device Collaborative Language Modeling via a Mixture of Generalists and Specialists](https://mlsanity.com/paper/2409.13931/) (2024)
- [On the Byzantine-Resilience of Distillation-Based Federated Learning](https://mlsanity.com/paper/2402.12265/) (2024)
- [Robust Federated Inference](https://mlsanity.com/paper/2510.00310/) (2025)
- [CollabEdit: Towards Non-destructive Collaborative Knowledge Editing](https://mlsanity.com/paper/2410.09508/) (2024)
- [A Kernel Perspective on Distillation-based Collaborative Learning](https://mlsanity.com/paper/2410.17592/) (2024)
- [Collaborative Prediction: To Join or To Disjoin Datasets](https://mlsanity.com/paper/2506.11271/) (2025)
- [Preference-driven Knowledge Distillation for Few-shot Node Classification](https://mlsanity.com/paper/2510.10116/) (2025)

## Abstract (arXiv)

We consider a collaborative learning setting where the goal of each agent is to improve their own model by leveraging the expertise of collaborators, in addition to their own training data. To facilitate the exchange of expertise among agents, we propose a distillation-based method leveraging shared unlabeled auxiliary data, which is pseudo-labeled by the collective. Central to our method is a trust weighting scheme that serves to adaptively weigh the influence of each collaborator on the pseudo-labels until a consensus on how to label the auxiliary data is reached. We demonstrate empirically that our collaboration scheme is able to significantly boost the performance of individual models in the target domain from which the auxiliary data is sampled. By design, our method adeptly accommodates heterogeneity in model architectures and substantially reduces communication overhead compared to typical collaborative learning methods. At the same time, it can provably mitigate the negative impact of bad models on the collective.