mlsanityData to 7 Oct 2026

Semantic-Syntactic Discrepancy in Images (SSDI): Learning Meaning and Order of Features from Natural Images

arXiv 2401.17515 ↗First posted 31 Jan 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

Despite considerable progress in image classification tasks, classification models seem unaffected by the images that significantly deviate from those that appear natural to human eyes. Specifically, while human perception can easily identify abnormal appearances or compositions in images, classification models overlook any alterations in the arrangement of object parts as long as they are present in any order, even if unnatural. Hence, this work exposes the vulnerability of having semantic and syntactic discrepancy in images (SSDI) in the form of corruptions that remove or shuffle image patches or present images in the form of puzzles. To address this vulnerability, we propose the concept of "image grammar", comprising "image semantics" and "image syntax". Image semantics pertains to the interpretation of parts or patches within an image, whereas image syntax refers to the arrangement of these parts to form a coherent object. We present a semi-supervised two-stage method for learning the image grammar of visual elements and environments solely from natural images. While the first stage learns the semantic meaning of individual object parts, the second stage learns how their relative arrangement constitutes an entire object. The efficacy of the proposed approach is then demonstrated by achieving SSDI detection rates ranging from 70% to 90% on corruptions generated from CelebA and SUN-RGBD datasets. Code is publicly available at: https://github.com/ChunTao1999/SSDI/

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 45 references that are in the map. First posted on arXiv 31 Jan 2024.

Builds on

35 of 45 references.

References · 35

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2017Mask R-CNNcited by 7,259
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  10. 2013Intriguing properties of neural networkscited by 5,114
  11. 2016Feature Pyramid Networks for Object Detectioncited by 5,021
  12. 2014Deep Learning Face Attributes in the Wildcited by 3,679
23 more
  1. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  2. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  3. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  4. 2019Panoptic Feature Pyramid Networkscited by 644
  5. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  7. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  8. 2020Visual Transformers: Token-based Image Representation and Processing for Computer Visioncited by 333
  9. 2021Enhancing the Transferability of Adversarial Attacks through Variance Tuningcited by 242
  10. 2020Understanding and Improving Fast Adversarial Trainingcited by 213
  11. 2021PiCIE: Unsupervised Semantic Segmentation using Invariance and Equivariance in Clusteringcited by 163
  12. 2021Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalizationcited by 141
  13. 2017A Review on Deep Learning Techniques Applied to Semantic Segmentationcited by 122
  14. 2020Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classificationcited by 99
  15. 2018RedNet: Residual Encoder-Decoder Network for indoor RGB-D Semantic Segmentationcited by 65
  16. 2022When Adversarial Training Meets Vision Transformers: Recipes from Training to Architecturecited by 64
  17. 2023DFormer: Rethinking RGBD Representation Learning for Semantic Segmentationcited by 58
  18. 2023Transferable Adversarial Attacks on Vision Transformers with Token Gradient Regularizationcited by 42
  19. 2021Understanding and Improving Robustness of Vision Transformers through Patch-based Negative Augmentationcited by 40
  20. 2022LGV: Boosting Adversarial Example Transferability from Large Geometric Vicinitycited by 39
  21. 2019Saccader: Improving Accuracy of Hard Attention Models for Visioncited by 35
  22. 2019Towards Robust Learning with Different Label Noise Distributionscited by 14
  23. 2023Training on Foveated Images Improves Robustness to Adversarial Attackscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024On Inherent Adversarial Robustness of Active Vision Systemscited by 1