# Parallel Successive Learning for Dynamic Distributed Model Training over Heterogeneous Wireless Networks

arXiv: https://arxiv.org/abs/2202.02947 · HTML: https://mlsanity.com/paper/2202.02947/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 61 references that are in the map.
- First posted on arXiv 7 Feb 2022.

## Builds on (most cited first, 30 of 31)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [CVXPY: A Python-Embedded Modeling Language for Convex Optimization](https://mlsanity.com/paper/1603.00943/) (2016)
- [Generative Adversarial Networks: An Overview](https://mlsanity.com/paper/1710.07035/) (2017)
- [Adaptive Federated Learning in Resource Constrained Edge Computing Systems](https://mlsanity.com/paper/1804.05271/) (2018)
- [A Joint Learning and Communications Framework for Federated Learning over Wireless Networks](https://mlsanity.com/paper/1909.07972/) (2019)
- [Client-Edge-Cloud Hierarchical Federated Learning](https://mlsanity.com/paper/1905.06641/) (2019)
- [Broadband Analog Aggregation for Low-Latency Federated Edge Learning (Extended Version)](https://mlsanity.com/paper/1812.11494/) (2018)
- [Federated Learning over Wireless Fading Channels](https://mlsanity.com/paper/1907.09769/) (2019)
- [Towards an Intelligent Edge: Wireless Communication Meets Machine Learning](https://mlsanity.com/paper/1809.00343/) (2018)
- [Federated Learning over Wireless Networks: Convergence Analysis and Resource Allocation](https://mlsanity.com/paper/1910.13067/) (2019)
- [Wireless Network Intelligence at the Edge](https://mlsanity.com/paper/1812.02858/) (2018)
- [Federated Learning with Cooperating Devices: A Consensus Approach for Massive IoT Networks](https://mlsanity.com/paper/1912.13163/) (2019)
- [Towards Flexible Device Participation in Federated Learning](https://mlsanity.com/paper/2006.06954/) (2020)
- [Semi-Decentralized Federated Learning with Cooperative D2D Local Model Aggregations](https://mlsanity.com/paper/2103.10481/) (2021)
- [Wide Compression: Tensor Ring Nets](https://mlsanity.com/paper/1802.09052/) (2018)
- [Wireless Communications for Collaborative Federated Learning](https://mlsanity.com/paper/2006.02499/) (2020)
- [Multi-Stage Hybrid Federated Learning over Large-Scale D2D-Enabled Fog Networks](https://mlsanity.com/paper/2007.09511/) (2020)
- [Device Sampling for Heterogeneous Federated Learning: Theory, Algorithms, and Implementation](https://mlsanity.com/paper/2101.00787/) (2021)
- [SparCML: High-Performance Sparse Communication for Machine Learning](https://mlsanity.com/paper/1802.08021/) (2018)
- [Decentralized Federated Learning via SGD over Wireless D2D Networks](https://mlsanity.com/paper/2002.12507/) (2020)
- [Network-Aware Optimization of Distributed Learning for Fog Computing](https://mlsanity.com/paper/2004.08488/) (2020)
- [Accelerating Minibatch Stochastic Gradient Descent using Stratified Sampling](https://mlsanity.com/paper/1405.3080/) (2014)
- [UAV-assisted Online Machine Learning over Multi-Tiered Networks: A Hierarchical Nested Personalized Federated Learning Approach](https://mlsanity.com/paper/2106.15734/) (2021)
- [Dynamic Sampling and Selective Masking for Communication-Efficient Federated Learning](https://mlsanity.com/paper/2003.09603/) (2020)
- [Coded Federated Learning](https://mlsanity.com/paper/2002.09574/) (2020)
- [Principled Deep Neural Network Training through Linear Programming](https://mlsanity.com/paper/1810.03218/) (2018)
- [Dynamic Federated Learning](https://mlsanity.com/paper/2002.08782/) (2020)

## Built on by (most cited first, 18 of 18)

- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [Performative Federated Learning: A Solution to Model-Dependent and Heterogeneous Distribution Shifts](https://mlsanity.com/paper/2305.05090/) (2023)
- [Latency Optimization for Blockchain-Empowered Federated Learning in Multi-Server Edge Computing](https://mlsanity.com/paper/2203.09670/) (2022)
- [Multi-Edge Server-Assisted Dynamic Federated Learning with an Optimized Floating Aggregation Point](https://mlsanity.com/paper/2203.13950/) (2022)
- [Online Federated Learning via Non-Stationary Detection and Adaptation amidst Concept Drift](https://mlsanity.com/paper/2211.12578/) (2022)
- [Delay-Aware Hierarchical Federated Learning](https://mlsanity.com/paper/2303.12414/) (2023)
- [Towards Cooperative Federated Learning over Heterogeneous Edge/Fog Networks](https://mlsanity.com/paper/2303.08361/) (2023)
- [Multi-Source to Multi-Target Decentralized Federated Domain Adaptation](https://mlsanity.com/paper/2304.12422/) (2023)
- [Embedding Alignment for Unsupervised Federated Learning via Smart Data Exchange](https://mlsanity.com/paper/2208.02856/) (2022)
- [Resource-Aware Hierarchical Federated Learning in Wireless Video Caching Networks](https://mlsanity.com/paper/2402.04216/) (2024)
- [Asynchronous Multi-Model Dynamic Federated Learning over Wireless Networks: Theory, Modeling, and Optimization](https://mlsanity.com/paper/2305.13503/) (2023)
- [Device Sampling and Resource Optimization for Federated Learning in Cooperative Edge Networks](https://mlsanity.com/paper/2311.04350/) (2023)
- [Digital Over-the-Air Federated Learning in Multi-Antenna Systems](https://mlsanity.com/paper/2302.14648/) (2023)
- [Elastic Federated Learning over Open Radio Access Network (O-RAN) for Concurrent Execution of Multiple Distributed Learning Tasks](https://mlsanity.com/paper/2305.02109/) (2023)
- [Resource-Aware Hierarchical Federated Learning for Video Caching in Wireless Networks](https://mlsanity.com/paper/2311.06918/) (2023)
- [Online-Score-Aided Federated Learning for Resource-Constrained Wireless Clients with Continual Data Arrival](https://mlsanity.com/paper/2408.05886/) (2024)
- [The Transition from Centralized Machine Learning to Federated Learning for Mental Health in Education: A Survey of Current Methods and Future Directions](https://mlsanity.com/paper/2501.11714/) (2025)
- [Unsupervised Federated Optimization at the Edge: D2D-Enabled Learning without Labels](https://mlsanity.com/paper/2404.09861/) (2024)

## Abstract (arXiv)

Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices, via iterative local updates (at devices) and global aggregations (at the server). In this paper, we develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions: (i) Network, allowing decentralized cooperation among the devices via device-to-device (D2D) communications. (ii) Heterogeneity, interpreted at three levels: (ii-a) Learning: PSL considers heterogeneous number of stochastic gradient descent iterations with different mini-batch sizes at the devices; (ii-b) Data: PSL presumes a dynamic environment with data arrival and departure, where the distributions of local datasets evolve over time, captured via a new metric for model/concept drift. (ii-c) Device: PSL considers devices with different computation and communication capabilities. (iii) Proximity, where devices have different distances to each other and the access point. PSL considers the realistic scenario where global aggregations are conducted with idle times in-between them for resource efficiency improvements, and incorporates data dispersion and model dispersion with local model condensation into FedL. Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning. We then propose network-aware dynamic model tracking to optimize the model learning vs. resource efficiency tradeoff, which we show is an NP-hard signomial programming problem. We finally solve this problem through proposing a general optimization solver. Our numerical results reveal new findings on the interdependencies between the idle times in-between the global aggregations, model/concept drift, and D2D cooperation configuration.