# FedMAX: Mitigating Activation Divergence for Accurate and Communication-Efficient Federated Learning

arXiv: https://arxiv.org/abs/2004.03657 · HTML: https://mlsanity.com/paper/2004.03657/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Apr 2020.

## Built on by (most cited first, 14 of 14)

- [Towards Building the Federated GPT: Federated Instruction Tuning](https://mlsanity.com/paper/2305.05644/) (2023)
- [Fed-CBS: A Heterogeneity-Aware Client Sampling Mechanism for Federated Learning via Class-Imbalance Reduction](https://mlsanity.com/paper/2209.15245/) (2022)
- [Efficient Low-rank Backpropagation for Vision Transformer Adaptation](https://mlsanity.com/paper/2309.15275/) (2023)
- [FedDKD: Federated Learning with Decentralized Knowledge Distillation](https://mlsanity.com/paper/2205.00706/) (2022)
- [Accelerating Federated Learning with a Global Biased Optimiser](https://mlsanity.com/paper/2108.09134/) (2021)
- [Deep Class Incremental Learning from Decentralized Data](https://mlsanity.com/paper/2203.05984/) (2022)
- [Demystifying the Effects of Non-Independence in Federated Learning](https://mlsanity.com/paper/2103.11226/) (2021)
- [Weight Divergence Driven Divide-and-Conquer Approach for Optimal Federated Learning from non-IID Data](https://mlsanity.com/paper/2106.14503/) (2021)
- [Communication Optimization in Large Scale Federated Learning using Autoencoder Compressed Weight Updates](https://mlsanity.com/paper/2108.05670/) (2021)
- [Federated Learning for Commercial Image Sources](https://mlsanity.com/paper/2507.12903/) (2025)
- [Decentralised Traffic Incident Detection via Network Lasso](https://mlsanity.com/paper/2402.18167/) (2024)
- [Federated Computing -- Survey on Building Blocks, Extensions and Systems](https://mlsanity.com/paper/2404.02779/) (2024)
- [Few-Shot Class-Incremental Learning with Non-IID Decentralized Data](https://mlsanity.com/paper/2409.11657/) (2024)
- [FedASAP: Activation Statistics-driven Structured Adaptive Pruning for Efficient Personalized Federated Learning for Lesion Segmentation on brain MRI](https://mlsanity.com/paper/2609.19042/) (2026)

## Abstract (arXiv)

In this paper, we identify a new phenomenon called activation-divergence which occurs in Federated Learning (FL) due to data heterogeneity (i.e., data being non-IID) across multiple users. Specifically, we argue that the activation vectors in FL can diverge, even if subsets of users share a few common classes with data residing on different devices. To address the activation-divergence issue, we introduce a prior based on the principle of maximum entropy; this prior assumes minimal information about the per-device activation vectors and aims at making the activation vectors of same classes as similar as possible across multiple devices. Our results show that, for both IID and non-IID settings, our proposed approach results in better accuracy (due to the significantly more similar activation vectors across multiple devices), and is more communication-efficient than state-of-the-art approaches in FL. Finally, we illustrate the effectiveness of our approach on a few common benchmarks and two large medical datasets.