# Shard the Gradient, Scale the Model: Serverless Federated Aggregation via Gradient Partitioning

arXiv: https://arxiv.org/abs/2604.22072 · HTML: https://mlsanity.com/paper/2604.22072/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 32 references that are in the map.
- First posted on arXiv 23 Apr 2026.

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Sparse Communication for Distributed Gradient Descent](https://mlsanity.com/paper/1704.05021/) (2017)
- [Evaluation of Deep Convolutional Nets for Document Image Classification and Retrieval](https://mlsanity.com/paper/1502.07058/) (2015)
- [Client-Edge-Cloud Hierarchical Federated Learning](https://mlsanity.com/paper/1905.06641/) (2019)
- [Peer-to-peer Federated Learning on Graphs](https://mlsanity.com/paper/1901.11173/) (2019)
- [Communication-efficient distributed SGD with Sketching](https://mlsanity.com/paper/1903.04488/) (2019)
- [FedAT: A High-Performance and Communication-Efficient Federated Learning System with Asynchronous Tiers](https://mlsanity.com/paper/2010.05958/) (2020)
- [Occupy the Cloud: Distributed Computing for the 99%](https://mlsanity.com/paper/1702.04024/) (2017)
- [Towards Demystifying Serverless Machine Learning Training](https://mlsanity.com/paper/2105.07806/) (2021)
- [FedLess: Secure and Scalable Federated Learning Using Serverless Computing](https://mlsanity.com/paper/2111.03396/) (2021)
- [Adaptive Aggregation For Federated Learning](https://mlsanity.com/paper/2203.12163/) (2022)
- [Exploring the Impact of Serverless Computing on Peer To Peer Training Machine Learning](https://mlsanity.com/paper/2309.14139/) (2023)
- [SPIRT: A Fault-Tolerant and Reliable Peer-to-Peer Serverless ML Training Architecture](https://mlsanity.com/paper/2309.14148/) (2023)
- [LIFL: A Lightweight, Event-driven Serverless Platform for Federated Learning](https://mlsanity.com/paper/2405.10968/) (2024)

## Abstract (arXiv)

Federated learning (FL) aggregation on serverless platforms faces a hard scalability ceiling: existing architectures (lambda-FL, LIFL) partition clients across aggregators, but every aggregator must hold the complete model gradient in memory. When gradients exceed the per-function memory limit (e.g., 10 GB on AWS Lambda), aggregation becomes infeasible regardless of tree depth or branching factor. We propose GradsSharding, which instead partitions the gradient tensor into M shards, each averaged independently by a serverless function that receives contributions from all clients. Because FedAvg averaging is element-wise, this produces bit-identical results to tree-based approaches, so model accuracy is invariant by construction. Per-function memory is bounded at O(|θ|/M), independent of client count, enabling aggregation of arbitrarily large models. We evaluate GradsSharding against lambda-FL and LIFL through HPC experiments and real AWS Lambda deployments across model sizes from 43 MB to 5 GB. Results show a cost crossover at approximately 500 MB gradient size, 2.7x cost reduction at VGG-16 scale, and that GradsSharding is the only architecture that remains deployable beyond the serverless memory ceiling.