Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning
Not on the map. Why?
Abstract (arXiv)
Split Federated Learning (SFL) enables collaborative training between resource-constrained edge devices and a compute-rich server. Communication overhead is a central issue in SFL and can be mitigated with auxiliary networks. Yet, the fundamental client-side computation challenge remains, as back-propagation requires substantial memory and computation costs, severely limiting the scale of models that edge devices can support. To enable more resource-efficient client computation and reduce the client-server communication, we propose HERON-SFL, a novel hybrid optimization framework that integrates zeroth-order (ZO) optimization for local client training while retaining first-order (FO) optimization on the server. With the assistance of auxiliary networks, ZO updates enable clients to approximate local gradients using perturbed forward-only evaluations per step, eliminating memory-intensive activation caching and avoiding explicit gradient computation in the traditional training process. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL's convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms. Empirically, on ResNet training and language model (LM) fine-tuning tasks, HERON-SFL matches benchmark accuracy while reducing client peak memory by up to 64% and client-side compute cost by up to 33% per step, substantially expanding the range of models that can be trained or adapted on resource-limited devices.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 45 of its 60 references that are in the map. First posted on arXiv 14 Jan 2026.
Builds on
45 of 60 references.
References · 45
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2019Advances and Open Problems in Federated Learningcited by 3,021
- 2020Adaptive Federated Optimizationcited by 1,036
- 2016Training Deep Nets with Sublinear Memory Costcited by 825
- 2017ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Modelscited by 656
33 more
- 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
- 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 391
- 2023Fine-Tuning Language Models with Just Forward Passescited by 337
- 2020SplitFed: When Federated Learning Meets Split Learningcited by 308
- 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
- 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
- 2018Distributed learning of deep neural network over multiple agentscited by 277
- 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
- 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 193
- 2020A Primer on Zeroth-Order Optimization in Signal Processing and Machine Learningcited by 160
- 2018Greedy Layerwise Learning Can Scale to ImageNetcited by 113
- 2018Zeroth-Order Stochastic Variance Reduction for Nonconvex Optimizationcited by 109
- 2019Decoupled Greedy Learning of CNNscited by 76
- 2023DeepZero: Scaling up Zeroth-Order Optimization for Deep Model Trainingcited by 76
- 2022Split Learning over Wireless Networks: Parallel Design and Resource Managementcited by 74
- 2024AdaptSFL: Adaptive Split Federated Learning in Resource-constrained Edge Networkscited by 64
- 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
- 2022Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learningcited by 59
- 2024Hierarchical Split Federated Learning: Convergence Analysis and System Optimizationcited by 38
- 2020TinyTL: Reduce Activations, Not Trainable Parameters for Efficient On-Device Learningcited by 32
- 2024The Future of Large Language Model Pre-training is Federatedcited by 30
- 2023Reducing Communication for Split Learning by Randomized Top-k Sparsificationcited by 22
- 2024On the Convergence of Zeroth-Order Federated Tuning for Large Language Modelscited by 18
- 2024Convergence Analysis of Split Federated Learning on Heterogeneous Datacited by 17
- 2024Achieving Dimension-Free Communication in Federated Learning via Zeroth-Order Optimizationcited by 12
- 2024Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inferencecited by 12
- 2022Desirable Companion for Vertical Federated Learning: New Zeroth-Order Gradient Based Algorithmcited by 9
- 2023Exploring the Privacy-Energy Consumption Tradeoff for Split Federated Learningcited by 9
- 2024Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networkscited by 7
- 2025FSL-SAGE: Accelerating Federated Split Learning via Smashed Activation Gradient Estimationcited by 7
- 2023All Rivers Run to the Sea: Private Learning with Asymmetric Flowscited by 4
- 2024SplitFedZip: Learned Compression for Data Transfer Reduction in Split-Federated Learningcited by 4
- 2025Federated Split Learning with Improved Communication and Storage Efficiencycited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.