# Mask-Encoded Sparsification: Mitigating Biased Gradients in Communication-Efficient Split Learning

arXiv: https://arxiv.org/abs/2408.13787 · HTML: https://mlsanity.com/paper/2408.13787/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 45 references that are in the map.
- First posted on arXiv 25 Aug 2024.

## Builds on (most cited first, 28 of 28)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networks](https://mlsanity.com/paper/1908.05033/) (2019)
- [The Convergence of Sparsified Gradient Methods](https://mlsanity.com/paper/1809.10505/) (2018)
- [Gradient Sparsification for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1710.09854/) (2017)
- [On Biased Compression for Distributed Learning](https://mlsanity.com/paper/2002.12410/) (2020)
- [Error Compensated Distributed SGD Can Be Accelerated](https://mlsanity.com/paper/2010.00091/) (2020)
- [Compressed-VFL: Communication-Efficient Learning with Vertically Partitioned Data](https://mlsanity.com/paper/2206.08330/) (2022)
- [EF21-P and Friends: Improved Theoretical Communication Complexity for Distributed Optimization with Bidirectional Compression](https://mlsanity.com/paper/2209.15218/) (2022)
- [DoCoFL: Downlink Compression for Cross-Device Federated Learning](https://mlsanity.com/paper/2302.00543/) (2023)
- [Optimal Clipping and Magnitude-aware Differentiation for Improved Quantization-aware Training](https://mlsanity.com/paper/2206.06501/) (2022)
- [Sketching for First Order Method: Efficient Algorithm for Low-Bandwidth Channel and Vulnerability](https://mlsanity.com/paper/2210.08371/) (2022)
- [Reducing Communication for Split Learning by Randomized Top-k Sparsification](https://mlsanity.com/paper/2305.18469/) (2023)
- [SplitGP: Achieving Both Generalization and Personalization in Federated Learning](https://mlsanity.com/paper/2212.08343/) (2022)
- [EF-BV: A Unified Theory of Error Feedback and Variance Reduction Mechanisms for Biased and Unbiased Compression in Distributed Optimization](https://mlsanity.com/paper/2205.04180/) (2022)
- [Federated Adversarial Learning: A Framework with Convergence Analysis](https://mlsanity.com/paper/2208.03635/) (2022)
- [Communication-Efficient Federated Learning for Heterogeneous Edge Devices Based on Adaptive Gradient Quantization](https://mlsanity.com/paper/2212.08272/) (2022)
- [Quantized Distributed Training of Large Models with Convergence Guarantees](https://mlsanity.com/paper/2302.02390/) (2023)
- [Predictive GAN-powered Multi-Objective Optimization for Hybrid Federated Split Learning](https://mlsanity.com/paper/2209.02428/) (2022)
- [Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence](https://mlsanity.com/paper/2302.13562/) (2023)
- [Performance of Large Language Models in a Computer Science Degree Program](https://mlsanity.com/paper/2308.02432/) (2023)

## Built on by (most cited first, 2 of 2)

- [LightSplit: Practical Privacy-Preserving Split Learning via Orthogonal Projections](https://mlsanity.com/paper/2605.13265/) (2026)
- [Importance-Aware Feature Sparsification for Wireless Split Learning](https://mlsanity.com/paper/2609.39194/) (2026)

## Abstract (arXiv)

This paper introduces a novel framework designed to achieve a high compression ratio in Split Learning (SL) scenarios where resource-constrained devices are involved in large-scale model training. Our investigations demonstrate that compressing feature maps within SL leads to biased gradients that can negatively impact the convergence rates and diminish the generalization capabilities of the resulting models. Our theoretical analysis provides insights into how compression errors critically hinder SL performance, which previous methodologies underestimate. To address these challenges, we employ a narrow bit-width encoded mask to compensate for the sparsification error without increasing the order of time complexity. Supported by rigorous theoretical analysis, our framework significantly reduces compression errors and accelerates the convergence. Extensive experiments also verify that our method outperforms existing solutions regarding training efficiency and communication complexity.