# A Closer Look at Codistillation for Distributed Training

arXiv: https://arxiv.org/abs/2010.02838 · HTML: https://mlsanity.com/paper/2010.02838/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 8 of 8)

- [Towards Model Agnostic Federated Learning Using Knowledge Distillation](https://mlsanity.com/paper/2110.15210/) (2021)
- [Personalized Federated Learning for Heterogeneous Clients with Clustered Knowledge Transfer](https://mlsanity.com/paper/2109.08119/) (2021)
- [lo-fi: distributed fine-tuning without communication](https://mlsanity.com/paper/2210.11948/) (2022)
- [Scalable Collaborative Learning via Representation Sharing](https://mlsanity.com/paper/2211.10943/) (2022)
- [Embracing Federated Learning: Enabling Weak Client Participation via Partial Model Training](https://mlsanity.com/paper/2406.15125/) (2024)
- [Learning from straggler clients in federated learning](https://mlsanity.com/paper/2403.09086/) (2024)
- [Leveraging Learning Metrics for Improved Federated Learning](https://mlsanity.com/paper/2309.00257/) (2023)
- [FedX: Adaptive Model Decomposition and Quantization for IoT Federated Learning](https://mlsanity.com/paper/2504.12849/) (2025)

## Abstract (arXiv)

Codistillation has been proposed as a mechanism to share knowledge among concurrently trained models by encouraging them to represent the same function through an auxiliary loss. This contrasts with the more commonly used fully-synchronous data-parallel stochastic gradient descent methods, where different model replicas average their gradients (or parameters) at every iteration and thus maintain identical parameters. We investigate codistillation in a distributed training setup, complementing previous work which focused on extremely large batch sizes. Surprisingly, we find that even at moderate batch sizes, models trained with codistillation can perform as well as models trained with synchronous data-parallel methods, despite using a much weaker synchronization mechanism. These findings hold across a range of batch sizes and learning rate schedules, as well as different kinds of models and datasets. Obtaining this level of accuracy, however, requires properly accounting for the regularization effect of codistillation, which we highlight through several empirical observations. Overall, this work contributes to a better understanding of codistillation and how to best take advantage of it in a distributed computing environment.