# Differentially Private CutMix for Split Learning with Vision Transformer

arXiv: https://arxiv.org/abs/2210.15986 · HTML: https://mlsanity.com/paper/2210.15986/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 38 references that are in the map.
- First posted on arXiv 28 Oct 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features](https://mlsanity.com/paper/1905.04899/) (2019)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Renyi Differential Privacy](https://mlsanity.com/paper/1702.07476/) (2017)
- [Intriguing Properties of Vision Transformers](https://mlsanity.com/paper/2105.10497/) (2021)
- [Split learning for health: Distributed deep learning without sharing raw patient data](https://mlsanity.com/paper/1812.00564/) (2018)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [SplitFed: When Federated Learning Meets Split Learning](https://mlsanity.com/paper/2004.12088/) (2020)
- [Distributed learning of deep neural network over multiple agents](https://mlsanity.com/paper/1810.06060/) (2018)
- [FedVision: An Online Visual Object Detection Platform Powered by Federated Learning](https://mlsanity.com/paper/2001.06202/) (2020)
- [Wireless Network Intelligence at the Edge](https://mlsanity.com/paper/1812.02858/) (2018)
- [Detailed comparison of communication efficiency of split learning and federated learning](https://mlsanity.com/paper/1909.09145/) (2019)
- [DP-InstaHide: Provably Defusing Poisoning and Backdoor Attacks with Differentially Private Data Augmentations](https://mlsanity.com/paper/2103.02079/) (2021)
- [FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks](https://mlsanity.com/paper/2111.11066/) (2021)
- [Differentially Private Label Protection in Split Learning](https://mlsanity.com/paper/2203.02073/) (2022)
- [Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Things](https://mlsanity.com/paper/2103.02762/) (2021)
- [Feature Space Hijacking Attacks against Differentially Private Split Learning](https://mlsanity.com/paper/2201.04018/) (2022)
- [Visual Transformer Meets CutMix for Improved Accuracy, Communication Efficiency, and Data Privacy in Split Learning](https://mlsanity.com/paper/2207.00234/) (2022)
- [AirMixML: Over-the-Air Data Mixup for Inherently Privacy-Preserving Edge Machine Learning](https://mlsanity.com/paper/2105.00395/) (2021)

## Built on by (most cited first, 13 of 13)

- [Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers](https://mlsanity.com/paper/2505.21497/) (2025)
- [PosterGen: Aesthetic-Aware Multi-Modal Paper-to-Poster Generation via Multi-Agent LLMs](https://mlsanity.com/paper/2508.17188/) (2025)
- [DP-Mix: Mixup-based Data Augmentation for Differentially Private Learning](https://mlsanity.com/paper/2311.01295/) (2023)
- [FeSViBS: Federated Split Learning of Vision Transformer with Block Sampling](https://mlsanity.com/paper/2306.14638/) (2023)
- [A Survey of Privacy Threats and Defense in Vertical Federated Learning: From Model Life Cycle Perspective](https://mlsanity.com/paper/2402.03688/) (2024)
- [Efficient Partitioning Vision Transformer on Edge Devices for Distributed Inference](https://mlsanity.com/paper/2410.11650/) (2024)
- [SLPerf: a Unified Framework for Benchmarking Split Learning](https://mlsanity.com/paper/2304.01502/) (2023)
- [Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMix](https://mlsanity.com/paper/2408.01040/) (2024)
- [SplitAMC: Split Learning for Robust Automatic Modulation Classification](https://mlsanity.com/paper/2304.12200/) (2023)
- [Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach](https://mlsanity.com/paper/2502.06355/) (2025)
- [Enhancing Privacy in ControlNet and Stable Diffusion via Split Learning](https://mlsanity.com/paper/2409.08503/) (2024)
- [MedSegNet10: A Publicly Accessible Network Repository for Split Federated Medical Image Segmentation](https://mlsanity.com/paper/2503.20830/) (2025)
- [Scalable and Private Federated Learning Using Distributed Differential Privacy and Secure Aggregation](https://mlsanity.com/paper/2604.07125/) (2026)

## Abstract (arXiv)

Recently, vision transformer (ViT) has started to outpace the conventional CNN in computer vision tasks. Considering privacy-preserving distributed learning with ViT, federated learning (FL) communicates models, which becomes ill-suited due to ViT' s large model size and computing costs. Split learning (SL) detours this by communicating smashed data at a cut-layer, yet suffers from data privacy leakage and large communication costs caused by high similarity between ViT' s smashed data and input data. Motivated by this problem, we propose DP-CutMixSL, a differentially private (DP) SL framework by developing DP patch-level randomized CutMix (DP-CutMix), a novel privacy-preserving inter-client interpolation scheme that replaces randomly selected patches in smashed data. By experiment, we show that DP-CutMixSL not only boosts privacy guarantees and communication efficiency, but also achieves higher accuracy than its Vanilla SL counterpart. Theoretically, we analyze that DP-CutMix amplifies R'enyi DP (RDP), which is upper-bounded by its Vanilla Mixup counterpart.