mlsanityData to 14 Oct 2026

Visual Transformer Meets CutMix for Improved Accuracy, Communication Efficiency, and Data Privacy in Split Learning

arXiv 2207.00234 ↗First posted 1 Jul 2022cs.LG cs.CR cs.CVcited by 8

Not on the map. Why?

Abstract (arXiv)

This article seeks for a distributed learning solution for the visual transformer (ViT) architectures. Compared to convolutional neural network (CNN) architectures, ViTs often have larger model sizes, and are computationally expensive, making federated learning (FL) ill-suited. Split learning (SL) can detour this problem by splitting a model and communicating the hidden representations at the split-layer, also known as smashed data. Notwithstanding, the smashed data of ViT are as large as and as similar as the input data, negating the communication efficiency of SL while violating data privacy. To resolve these issues, we propose a new form of CutSmashed data by randomly punching and compressing the original smashed data. Leveraging this, we develop a novel SL framework for ViT, coined CutMixSL, communicating CutSmashed data. CutMixSL not only reduces communication costs and privacy leakage, but also inherently involves the CutMix data augmentation, improving accuracy and scalability. Simulations corroborate that CutMixSL outperforms baselines such as parallelized SL and SplitFed that integrates FL with SL.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 31 of its 44 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

31 of 44 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020End-to-End Object Detection with Transformerscited by 7,414
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
  10. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  11. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,475
  12. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
19 more
  1. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  2. 2021Transformers in Vision: A Surveycited by 1,013
  3. 2020A Survey on Visual Transformercited by 654
  4. 2021How to train your ViT? Data, Augmentation, and Regularization in Vision Transformerscited by 569
  5. 2021Intriguing Properties of Vision Transformerscited by 446
  6. 2018Split learning for health: Distributed deep learning without sharing raw patient datacited by 395
  7. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  8. 2020SplitFed: When Federated Learning Meets Split Learningcited by 310
  9. 2018Distributed learning of deep neural network over multiple agentscited by 279
  10. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  11. 2015Federated Optimization:Distributed Optimization Beyond the Datacentercited by 195
  12. 2021Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learningcited by 111
  13. 2021Revisiting Locally Supervised Learning: an Alternative to End-to-end Trainingcited by 71
  14. 2020NoPeek: Information leakage reduction to share activations in distributed deep learningcited by 66
  15. 2020Communication-Efficient and Distributed Learning Over Wireless Networks: Principles and Applicationscited by 62
  16. 2021Server-Side Local Gradient Averaging and Learning Rate Acceleration for Scalable Split Learningcited by 23
  17. 2021Evaluation and Optimization of Distributed Machine Learning Techniques for Internet of Thingscited by 20
  18. 2020FMix: Enhancing Mixed Sample Data Augmentationcited by 11
  19. 2021Federated Learning with Dynamic Transformer for Text to Speechcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 2; 2025: 2; 2026: 1.

8 papers.

  1. 2022Differentially Private CutMix for Split Learning with Vision Transformercited by 13
  2. 2023SLPerf: a Unified Framework for Benchmarking Split Learningcited by 3
  3. 2023Edge-assisted U-Shaped Split Federated Learning with Privacy-preserving for Internet of Thingscited by 3
  4. 2024Privacy-Preserving Split Learning with Vision Transformers using Patch-Wise Random and Noisy CutMixcited by 2
  5. 2024Training Strategies for Isolated Sign Language Recognitioncited by 1
  6. 2025Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approachcited by 1
  7. 2025MedSegNet10: A Publicly Accessible Network Repository for Split Federated Medical Image Segmentationcited by 0
  8. 2026Don't Send What You Don't Need: Question-Guided Token Pruning as a Privacy Defense for Vision-Language Modelscited by 0