Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement
Not on the map. Why?
Abstract (arXiv)
The attention mechanism has been proven effective on various visual tasks in recent years. In the semantic segmentation task, the attention mechanism is applied in various methods, including the case of both Convolution Neural Networks (CNN) and Vision Transformer (ViT) as backbones. However, we observe that the attention mechanism is vulnerable to patch-based adversarial attacks. Through the analysis of the effective receptive field, we attribute it to the fact that the wide receptive field brought by global attention may lead to the spread of the adversarial patch. To address this issue, in this paper, we propose a Robust Attention Mechanism (RAM) to improve the robustness of the semantic segmentation model, which can notably relieve the vulnerability against patch-based attacks. Compared to the vallina attention mechanism, RAM introduces two novel modules called Max Attention Suppression and Random Attention Dropout, both of which aim to refine the attention matrix and limit the influence of a single adversarial patch on the semantic segmentation results of other positions. Extensive experiments demonstrate the effectiveness of our RAM to improve the robustness of semantic segmentation models against various patch-based attack methods under different attack settings.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 70 of its 78 references that are in the map. First posted on arXiv 3 Jan 2024.
Builds on
70 of 78 references.
References · 70
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
- 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,415
- 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
- 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
- 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
- 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
- 2022A ConvNet for the 2020scited by 3,579
- 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,241
- 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
- 2016Pyramid Scene Parsing Networkcited by 2,479
38 more
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,351
- 2017Non-local Neural Networkscited by 2,293
- 2017Rethinking Atrous Convolution for Semantic Image Segmentationcited by 2,161
- 2020Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attackscited by 1,451
- 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,369
- 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
- 2021Per-Pixel Classification is Not All You Need for Semantic Segmentationcited by 1,152
- 2021EfficientNetV2: Smaller Models and Faster Trainingcited by 1,135
- 2017Learning Transferable Architectures for Scalable Image Recognitioncited by 1,101
- 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 971
- 2021Segmenter: Transformer for Semantic Segmentationcited by 855
- 2017SphereFace: Deep Hypersphere Embedding for Face Recognitioncited by 737
- 2018CCNet: Criss-Cross Attention for Semantic Segmentationcited by 718
- 2019Square Attack: a query-efficient black-box adversarial attack via random searchcited by 685
- 2019Panoptic Feature Pyramid Networkscited by 645
- 2014Semantic Image Segmentation with Deep Convolutional Nets and Fully Connected CRFscited by 554
- 2017Understanding the Effective Receptive Field in Deep Convolutional Neural Networkscited by 553
- 2021MetaFormer Is Actually What You Need for Visioncited by 524
- 2017Adversarial Patchcited by 517
- 2019Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Comingcited by 336
- 2019Minimally distorted Adversarial Examples with a Fast Adaptive Boundary Attackcited by 334
- 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
- 2021Understanding Robustness of Transformers for Image Classificationcited by 249
- 2021Are Transformers More Robust Than CNNs?cited by 187
- 2021On the Robustness of Vision Transformers to Adversarial Examplescited by 167
- 2018DPatch: An Adversarial Patch Attack on Object Detectorscited by 147
- 2019Expectation-Maximization Attention Networks for Semantic Segmentationcited by 147
- 2018OCNet: Object Context Network for Scene Parsingcited by 140
- 2021Towards Robust Vision Transformercited by 137
- 2021On the Convergence and Robustness of Adversarial Trainingcited by 130
- 2018LaVAN: Localized and Visible Adversarial Noisecited by 113
- 2019Benchmarking the Robustness of Semantic Segmentation Modelscited by 111
- 2021On the Adversarial Robustness of Vision Transformerscited by 106
- 2019On Physical Adversarial Patches for Object Detectioncited by 93
- 2021Adversarial Sticker: A Stealthy Attack Method in the Physical Worldcited by 93
- 2021Improving Transferability of Adversarial Patches on Face Recognition with Generative Modelscited by 80
- 2022Improving Vision Transformers by Revisiting High-frequency Componentscited by 76
- 2020PatchAttack: A Black-box Texture-based Attack with Reinforcement Learningcited by 67
50 most cited of 70 All 70 (JSON)
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.