mlsanityData to 14 Oct 2026

One Leaf Reveals the Season: Occlusion-Based Contrastive Learning with Semantic-Aware Views for Efficient Visual Representation

arXiv 2411.09858 ↗First posted 15 Nov 2024cs.CVcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper proposes a scalable and straightforward pre-training paradigm for efficient visual conceptual representation called occluded image contrastive learning (OCL). Our OCL approach is simple: we randomly mask patches to generate different views within an image and contrast them among a mini-batch of images. The core idea behind OCL consists of two designs. First, masked tokens have the potential to significantly diminish the conceptual redundancy inherent in images, and create distinct views with substantial fine-grained differences on the semantic concept level instead of the instance level. Second, contrastive learning is adept at extracting high-level semantic conceptual features during the pre-training, circumventing the high-frequency interference and additional costs associated with image reconstruction. Importantly, OCL learns highly semantic conceptual representations efficiently without relying on hand-crafted data augmentations or additional auxiliary modules. Empirically, OCL demonstrates high scalability with Vision Transformers, as the ViT-L/16 can complete pre-training in 133 hours using only 4 A100 GPUs, achieving 85.8% accuracy in downstream fine-tuning tasks. Code is available at https://anonymous.4open.science/r/OLRS/.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 48 of its 56 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

48 of 56 references.

References · 48

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  5. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  7. 2017Mask R-CNNcited by 7,277
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  9. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  12. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
36 more
  1. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  2. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  3. 2019Natural Adversarial Examplescited by 1,408
  4. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  5. 2018Unified Perceptual Parsing for Scene Understandingcited by 1,271
  6. 2021SimMIM: A Simple Framework for Masked Image Modelingcited by 1,153
  7. 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
  8. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  9. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  10. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  11. 2020Dense Contrastive Learning for Self-Supervised Visual Pre-Trainingcited by 512
  12. 2022Scaling Language-Image Pre-training via Maskingcited by 317
  13. 2022Masked Siamese Networks for Label-Efficient Learningcited by 302
  14. 2022Context Autoencoder for Self-Supervised Representation Learningcited by 297
  15. 2023Contrast with Reconstruct: Contrastive 3D Representation Learning Guided by Generative Pretrainingcited by 164
  16. 2021Efficient Self-supervised Vision Transformers for Representation Learningcited by 149
  17. 2022Contrastive Masked Autoencoders are Stronger Vision Learnerscited by 133
  18. 2021MST: Masked Self-Supervised Transformer for Visual Representationcited by 114
  19. 2022Efficient Self-supervised Learning with Contextualized Target Representations for Vision, Speech and Languagecited by 110
  20. 2022How Mask Matters: Towards Theoretical Understandings of Masked Autoencoderscited by 86
  21. 2022Siamese Image Modeling for Self-Supervised Vision Representation Learningcited by 78
  22. 2022Green Hierarchical Vision Transformer for Masked Image Modelingcited by 50
  23. 2022MixMAE: Mixed and Masked Autoencoder for Efficient Pretraining of Hierarchical Vision Transformerscited by 47
  24. 2022Masked Image Modeling with Denoising Contrastcited by 45
  25. 2023Understanding Masked Autoencoders via Hierarchical Latent Variable Modelscited by 39
  26. 2021Self-Supervised Visual Representations Learning by Contrastive Mask Predictioncited by 31
  27. 2022A simple, efficient and scalable contrastive masked autoencoder for learning visual representationscited by 25
  28. 2022Attentive Mask CLIPcited by 23
  29. 2022Extreme Masking for Learning Instance and Distributed Visual Representationscited by 21
  30. 2024Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learningcited by 20
  31. 2023DropPos: Pre-Training Vision Transformers by Reconstructing Dropped Positionscited by 19
  32. 2023Layer Grafted Pre-training: Bridging Contrastive Learning And Masked Image Modeling For Label-Efficient Representationscited by 18
  33. 2023Mimic before Reconstruct: Enhancing Masked Autoencoders with Feature Mimickingcited by 12
  34. 2022ConCL: Concept Contrastive Learning for Dense Prediction Pre-training in Pathology Imagescited by 11
  35. 2023CoMAE: Single Model Hybrid Pre-training on Small-Scale RGB-D Datasetscited by 8
  36. 2023Enhancing Visual Grounding and Generalization: A Multi-Task Cycle Training Approach for Vision-Language Modelscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 13.

16 papers.

Sort:

By area:

  1. 2025Fewer Tokens, Greater Scaling: Self-Adaptive Visual Bases for Efficient and Expansive Representation Learningcited by 16
  2. 2026The Model Knows Which Tokens Matter: Automatic Token Selection via Noise Gatingcited by 11
  3. 2026Efficient Chest X-ray Representation Learning via Semantic-Partitioned Contrastive Learningcited by 10
  4. 2025Resilient Contrastive Pre-training under Non-Stationary Driftcited by 7
  5. 2026Multimodal Model for Computational Pathology:Representation Learning and Image Compressioncited by 6
  6. 2026A unified multi-task framework enables interpretable chest radiograph analysiscited by 6
  7. 2026XrayClaw: Cooperative-Competitive Multi-Agent Alignment for Trustworthy Chest X-ray Diagnosiscited by 5
  8. 2026Stepwise Token Selection for Efficient Multimodal Large Language Modelscited by 5
  9. 2025Bayesian Ego-graph Inference for Networked Multi-Agent Reinforcement Learningcited by 4
  10. 2026Towards Robust Endogenous Reasoning: Unifying Drift Adaptation in Non-Stationary Tuningcited by 2
  11. 2026PathSelect: Sequential Token Selection for Whole Slide Pathologycited by 2
  12. 2026Decoupling semantics from vision: A framework for faithful visual-text compression evaluationcited by 2
4 more
  1. 2026DiffPrune: differentiable information throttling for token pruning in vision-language modelscited by 1
  2. 2026From Snapshots to Symphonies: The Evolution of Protein Prediction from Static Structures to Generative Dynamics and Multimodal Interactionscited by 0
  3. 2026Autonomous Drift Learning in Data Streams: A Unified Perspectivecited by 0
  4. 2026StepPrune: Adaptive Sequential Visual Token Selection across Multimodal Large Language Modelscited by 0