mlsanityData to 7 Oct 2026

Breaking the Memory Wall for Heterogeneous Federated Learning via Model Splitting

arXiv 2410.11577 ↗First posted 12 Oct 2024cs.DC cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) enables multiple devices to collaboratively train a shared model while preserving data privacy. Ever-increasing model complexity coupled with limited memory resources on the participating devices severely bottlenecks the deployment of FL in real-world scenarios. Thus, a framework that can effectively break the memory wall while jointly taking into account the hardware and statistical heterogeneity in FL is urgently required. In this paper, we propose SmartSplit, a framework that effectively reduces the memory footprint on the device side while guaranteeing the training progress and model accuracy for heterogeneous FL through model splitting.Towards this end, SmartSplit employs a hierarchical structure to adaptively guide the overall training process. In each training round, the central manager, hosted on the server, dynamically selects the participating devices and sets the cutting layer by jointly considering the memory budget, training capacity, and data distribution of each device. The MEC manager, deployed within the edge server, proceeds to split the local model and perform training of the server-side portion. Meanwhile, it fine-tunes the splitting points based on the time-evolving statistical importance. The on-device manager, embedded inside each mobile device, continuously monitors the local training status while employing cost-aware checkpointing to match the runtime dynamic memory budget. Extensive experiments on representative datasets are conducted on both commercial off-the-shelf mobile device testbeds. The experimental results show that SmartSplit excels in FL training on highly memory-constrained mobile SoCs, offering up to a 94% peak latency reduction and 100-fold memory savings. It enhances accuracy performance by 1.49%-57.18% and adaptively adjusts to dynamic memory budgets through cost-aware recomputation.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 37 of its 75 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

37 of 75 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  11. 2018A Tutorial on Bayesian Optimizationcited by 734
  12. 2021Federated Learning Based on Dynamic Regularizationcited by 555
25 more
  1. 2017Differentially Private Federated Learning: A Client Level Perspectivecited by 451
  2. 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
  3. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 253
  4. 2019Model Pruning Enables Efficient Federated Learning on Edge Devicescited by 219
  5. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  6. 2022On-Device Training Under 256KB Memorycited by 158
  7. 2017Apprentice: Using Knowledge Distillation Techniques To Improve Low-Precision Network Accuracycited by 100
  8. 2020TiFL: A Tier-based Federated Learning Systemcited by 95
  9. 2019Detailed comparison of communication efficiency of split learning and federated learningcited by 76
  10. 2020End-to-End Evaluation of Federated Learning and Split Learning for Internet of Thingscited by 64
  11. 2019Low-Memory Neural Network Training: A Technical Reportcited by 56
  12. 2018SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networkscited by 55
  13. 2022FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clientscited by 46
  14. 2022Machine Learning Model Sizes and the Parameter Gapcited by 39
  15. 2021Differentially Private Federated Bayesian Optimization with Distributed Explorationcited by 32
  16. 2021FedAdapt: Adaptive Offloading for IoT Devices in Federated Learningcited by 26
  17. 2020FedSL: Federated Split Learning on Distributed Sequential Data in Recurrent Neural Networkscited by 23
  18. 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 22
  19. 2023FedVS: Straggler-Resilient and Privacy-Preserving Vertical Federated Learning for Split Modelscited by 17
  20. 2024Breaking the Memory Wall for Heterogeneous Federated Learning via Progressive Trainingcited by 10
  21. 2024Heterogeneity-Aware Memory Efficient Federated Learning via Progressive Layer Freezingcited by 10
  22. 2024FedGCS: A Generative Framework for Efficient Client Selection in Federated Learning via Gradient-based Optimizationcited by 9
  23. 2024Bridging Memory Gaps: Scaling Federated Learning for Heterogeneous Clientscited by 3
  24. 2021Memory at Your Service: Fast Memory Allocation for Latency-critical Servicescited by 1
  25. 2024Heterogeneity-Aware Coordination for Federated Learning via Stitching Pre-trained blockscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Learning Like Humans: Resource-Efficient Federated Fine-Tuning through Cognitive Developmental Stagescited by 4
  2. 2026Floe: Federated Specialization for Real-Time LLM-SLM Inferencecited by 2
  3. 2025Ampere: Communication-Efficient and High-Accuracy Split Federated Learningcited by 1
  4. 2026When Is Shallow Enough? Adaptive Split Federated Learning with Client-Specific Sufficiency Estimationcited by 0