# Communication Efficient Split Learning of ViTs with Attention-based Double Compression

arXiv: https://arxiv.org/abs/2509.15058 · HTML: https://mlsanity.com/paper/2509.15058/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 40 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Builds on (most cited first, 30 of 30)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Vision Transformers Need Registers](https://mlsanity.com/paper/2309.16588/) (2023)
- [Quantifying Attention Flow in Transformers](https://mlsanity.com/paper/2005.00928/) (2020)
- [Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations](https://mlsanity.com/paper/2202.07800/) (2022)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Split Computing and Early Exiting for Deep Learning Applications: Survey and Research Challenges](https://mlsanity.com/paper/2103.04505/) (2021)
- [UVeQFed: Universal Vector Quantization for Federated Learning](https://mlsanity.com/paper/2006.03262/) (2020)
- [BottleNet++: An End-to-End Approach for Feature Compression in Device-Edge Co-Inference Systems](https://mlsanity.com/paper/1910.14315/) (2019)
- [Split Learning in 6G Edge Networks](https://mlsanity.com/paper/2306.12194/) (2023)
- [NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformers](https://mlsanity.com/paper/2211.16056/) (2022)
- [EasyQuant: Post-training Quantization via Scale Optimization](https://mlsanity.com/paper/2006.16669/) (2020)
- [Teaching Matters: Investigating the Role of Supervision in Vision Transformers](https://mlsanity.com/paper/2212.03862/) (2022)
- [PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformers](https://mlsanity.com/paper/2209.05687/) (2022)
- [JALAD: Joint Accuracy- and Latency-Aware Deep Structure Decoupling for Edge-Cloud Execution](https://mlsanity.com/paper/1812.10027/) (2018)
- [Edge-Host Partitioning of Deep Neural Networks with Feature Space Encoding for Resource-Constrained Internet-of-Things Platforms](https://mlsanity.com/paper/1802.03835/) (2018)
- [Supervised Compression for Resource-Constrained Edge Computing Systems](https://mlsanity.com/paper/2108.11898/) (2021)
- [Analyzing Vision Transformers for Image Classification in Class Embedding Space](https://mlsanity.com/paper/2310.18969/) (2023)
- [A Federated Learning Framework for Healthcare IoT devices](https://mlsanity.com/paper/2005.05083/) (2020)
- [Reducing Communication for Split Learning by Randomized Top-k Sparsification](https://mlsanity.com/paper/2305.18469/) (2023)
- [PowerQuant: Automorphism Search for Non-Uniform Quantization](https://mlsanity.com/paper/2301.09858/) (2023)
- [Adaptive Semantic Token Communication for Transformer-based Edge Inference](https://mlsanity.com/paper/2505.17604/) (2025)
- [Prompt-CAM: Making Vision Transformers Interpretable for Fine-Grained Analysis](https://mlsanity.com/paper/2501.09333/) (2025)
- [FedVQCS: Federated Learning via Vector Quantized Compressed Sensing](https://mlsanity.com/paper/2204.07692/) (2022)
- [C3-SL: Circular Convolution-Based Batch-Wise Compression for Communication-Efficient Split Learning](https://mlsanity.com/paper/2207.12397/) (2022)
- [FAT: Learning Low-Bitwidth Parametric Representation via Frequency-Aware Transformation](https://mlsanity.com/paper/2102.07444/) (2021)
- [Enabling Edge Artificial Intelligence via Goal-oriented Deep Neural Network Splitting](https://mlsanity.com/paper/2312.03555/) (2023)
- [Distributed Constrained Optimization with Delayed Subgradient Information over Time-Varying Network under Adaptive Quantization](https://mlsanity.com/paper/2106.08271/) (2021)

## Built on by (most cited first, 2 of 2)

- [AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning](https://mlsanity.com/paper/2607.17913/) (2026)
- [Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Models](https://mlsanity.com/paper/2609.15671/) (2026)

## Abstract (arXiv)

This paper proposes a novel communication-efficient Split Learning (SL) framework, named Attention-based Double Compression (ADC), which reduces the communication overhead required for transmitting intermediate Vision Transformers activations during the SL training process. ADC incorporates two parallel compression strategies. The first one merges samples' activations that are similar, based on the average attention score calculated in the last client layer; this strategy is class-agnostic, meaning that it can also merge samples having different classes, without losing generalization ability nor decreasing final results. The second strategy follows the first and discards the least meaningful tokens, further reducing the communication cost. Combining these strategies not only allows for sending less during the forward pass, but also the gradients are naturally compressed, allowing the whole model to be trained without additional tuning or approximations of the gradients. Simulation results demonstrate that Attention-based Double Compression outperforms state-of-the-art SL frameworks by significantly reducing communication overheads while maintaining high accuracy.