mlsanityData to 14 Oct 2026

A Unified Masked Jigsaw Puzzle Framework for Vision and Language Models

arXiv 2601.12051 ↗First posted 17 Jan 2026cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

In federated learning, Transformer, as a popular architecture, faces critical challenges in defending against gradient attacks and improving model performance in both Computer Vision (CV) and Natural Language Processing (NLP) tasks. It has been revealed that the gradient of Position Embeddings (PEs) in Transformer contains sufficient information, which can be used to reconstruct the input data. To mitigate this issue, we introduce a Masked Jigsaw Puzzle (MJP) framework. MJP starts with random token shuffling to break the token order, and then a learnable textitunknown (unk) position embedding is used to mask out the PEs of the shuffled tokens. In this manner, the local spatial information which is encoded in the position embeddings is disrupted, and the models are forced to learn feature representations that are less reliant on the local spatial information. Notably, with the careful use of MJP, we can not only improve models' robustness against gradient attacks, but also boost their performance in both vision and text application scenarios, such as classification for images (textite.g., ImageNet-1K) and sentiment analysis for text (textite.g., Yelp and Amazon). Experimental results suggest that MJP is a unified framework for different Transformer-based models in both vision and language tasks. Code is publicly available via https://github.com/ywxsuperstar/transformerattack

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 49 of its 77 references that are in the map. First posted on arXiv 17 Jan 2026.

Builds on

49 of 77 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  9. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  10. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,422
37 more
  1. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  2. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  3. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,962
  4. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,887
  5. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  6. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
  7. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  8. 2018Self-Attention with Relative Position Representationscited by 1,120
  9. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,069
  10. 2019Deep Leakage from Gradientscited by 1,021
  11. 2021Transformers in Vision: A Surveycited by 1,013
  12. 2018Neural Network Acceptability Judgmentscited by 958
  13. 2020Pre-Trained Image Processing Transformercited by 893
  14. 2021PVT v2: Improved Baselines with Pyramid Vision Transformercited by 751
  15. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  16. 2017Convolutional Sequence to Sequence Learningcited by 552
  17. 2022PETR: Position Embedding Transformation for Multi-View 3D Object Detectioncited by 552
  18. 2016Neural Architectures for Named Entity Recognitioncited by 550
  19. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  20. 2020End-to-End Video Instance Segmentation with Transformerscited by 443
  21. 2021TransReID: Transformer-based Object Re-Identificationcited by 435
  22. 2021Conditional Positional Encodings for Vision Transformerscited by 406
  23. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  24. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  25. 2018Video Action Transformer Networkcited by 316
  26. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216
  27. 2021You Only Look at One Sequence: Rethinking Transformer in Vision through Object Detectioncited by 170
  28. 2022Self-Supervised Transformers for Unsupervised Object Discovery using Normalized Cutcited by 167
  29. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  30. 2021Transformer-Based Attention Networks for Continuous Pixel-Wise Predictioncited by 125
  31. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  32. 2020Provable Defense against Privacy Leakage in Federated Learning from Representation Perspectivecited by 94
  33. 2021TAG: Gradient Attack on Transformer-based Language Modelscited by 63
  34. 2022GradViT: Gradient Inversion of Vision Transformerscited by 53
  35. 2021SHAPE: Shifted Absolute Position Embedding for Transformerscited by 37
  36. 2022Masked Jigsaw Puzzle: A Versatile Position Embedding for Vision Transformerscited by 33
  37. 2020Privacy-preserving Collaborative Learning with Automatic Transformation Searchcited by 31

Built on by

Not cited yet.