Addressing Label Shift in Distributed Learning via Entropy Regularization
Not on the map. Why?
Abstract (arXiv)
We address the challenge of minimizing true risk in multi-node distributed learning. These systems are frequently exposed to both inter-node and intra-node label shifts, which present a critical obstacle to effectively optimizing model performance while ensuring that data remains confined to each node. To tackle this, we propose the Versatile Robust Label Shift (VRLS) method, which enhances the maximum likelihood estimation of the test-to-train label density ratio. VRLS incorporates Shannon entropy-based regularization and adjusts the density ratio during training to better handle label shifts at the test time. In multi-node learning environments, VRLS further extends its capabilities by learning and adapting density ratios across nodes, effectively mitigating label shifts and improving overall model performance. Experiments conducted on MNIST, Fashion MNIST, and CIFAR-10 demonstrate the effectiveness of VRLS, outperforming baselines by up to 20% in imbalanced settings. These results highlight the significant improvements VRLS offers in addressing label shifts. Our theoretical analysis further supports this by establishing high-probability bounds on estimation errors.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 43 of its 74 references that are in the map. First posted on arXiv 4 Feb 2025.
Builds on
43 of 74 references.
References · 43
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2017Federated Multi-Task Learningcited by 669
- 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
- 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
- 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
- 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
- 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
- 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
31 more
- 2018What is the Effect of Importance Weighting in Deep Learning?cited by 250
- 2018Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shiftcited by 240
- 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
- 2019Adaptive Gradient-Based Meta-Learning Methodscited by 183
- 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
- 2020Federated Learning with Compression: Unified Analysis and Sharp Guaranteescited by 159
- 2020Is Local SGD Better than Minibatch SGD?cited by 151
- 2020A Unified View of Label Shift Estimationcited by 119
- 2019Regularized Learning for Domain Adaptation under Label Shiftscited by 118
- 2020Rethinking Importance Weighting for Deep Learning under Distribution Shiftcited by 92
- 2023High Probability Convergence of Stochastic Gradient Methodscited by 48
- 2020Adaptive Gradient Quantization for Data-Parallel SGDcited by 43
- 2021Sharp Bounds for Federated Averaging (Local SGD) and Continuous Perspectivecited by 38
- 2022Faster Adaptive Federated Learningcited by 35
- 2022Multi-Level Branched Regularization for Federated Learningcited by 34
- 2022Domain Adaptation under Open Set Label Shiftcited by 32
- 2023RLSbench: Domain Adaptation Under Relaxed Label Shiftcited by 30
- 2020Optimal Bounds between f-Divergences and Integral Probability Metricscited by 27
- 2023GradMA: A Gradient-Memory-based Accelerated Federated Learning with Alleviated Catastrophic Forgettingcited by 27
- 2022Mitigating Data Heterogeneity in Federated Learning with Data Augmentationcited by 21
- 2019NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantizationcited by 17
- 2021Optimization-based Causal Estimation from Heterogenous Environmentscited by 16
- 2022FL Games: A Federated Learning Framework for Distribution Shiftscited by 14
- 2022Domain Adaptation under Missingness Shiftcited by 11
- 2023Minimum-Risk Recalibration of Classifierscited by 10
- 2022Unsupervised Learning under Latent Label Shiftcited by 9
- 2023Federated Learning under Covariate Shifts with Generalization Guaranteescited by 6
- 2023MaxEnt Loss: Constrained Maximum Entropy for Calibration under Out-of-Distribution Shiftcited by 6
- 2022Generalized Label Shift Correction via Minimum Uncertainty Principle: Theory and Algorithmcited by 3
- 2023On the Variance, Admissibility, and Stability of Empirical Risk Minimizationcited by 3
- 2023Distributed Extra-gradient with Optimal Complexity and Communication Guaranteescited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.