mlsanityData to 7 Oct 2026

Learning Visual Representations via Language-Guided Sampling

arXiv 2302.12248 ↗First posted 23 Feb 2023cs.CVcited by 32

Not on the map. Why?

Abstract (arXiv)

Although an object may appear in numerous contexts, we often describe it in a limited number of ways. Language allows us to abstract away visual variation to represent and communicate concepts. Building on this intuition, we propose an alternative approach to visual representation learning: using language similarity to sample semantically similar image pairs for contrastive learning. Our approach diverges from image-based contrastive learning by sampling view pairs using language similarity instead of hand-crafted augmentations or learned clusters. Our approach also differs from image-text contrastive learning by relying on pre-trained language models to guide the learning rather than directly minimizing a cross-modal loss. Through a series of experiments, we show that language-guided learning yields better features than image-based and image-text representation learning approaches.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 90 of its 124 references that are in the map. First posted on arXiv 23 Feb 2023.

Builds on

90 of 124 references.

References · 90

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  12. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  2. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  5. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  8. 2015VQA: Visual Question Answeringcited by 2,489
  9. 2017Billion-scale similarity search with GPUscited by 2,477
  10. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  11. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  12. 2013Describing Textures in the Wildcited by 2,110
  13. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  14. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  15. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  16. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  17. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  18. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  19. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  20. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  21. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  22. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  23. 2019Natural Adversarial Examplescited by 1,401
  24. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  25. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  26. 2016Colorful Image Colorizationcited by 1,113
  27. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  28. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  29. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  30. 2021Scaling Vision Transformerscited by 993
  31. 2021LoFTR: Detector-Free Local Feature Matching with Transformerscited by 974
  32. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  33. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  34. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  35. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  36. 2018Do Better ImageNet Models Transfer Better?cited by 560
  37. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  38. 2022GroupViT: Semantic Segmentation Emerges from Text Supervisioncited by 529

50 most cited of 90 All 90 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 11; 2025: 9; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Scalable 3D Captioning with Pretrained Modelscited by 201
  2. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  3. 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
  4. 2023Hyperbolic Image-Text Representationscited by 132
  5. 2024UniBind: LLM-Augmented Unified and Balanced Representation Space to Bind Them Allcited by 49
  6. 2023A General-Purpose Self-Supervised Model for Computational Pathologycited by 42
  7. 2024What to align in multimodal contrastive learning?cited by 41
  8. 2023If at First You Don't Succeed, Try, Try Again: Faithful Diffusion-based Text-to-Image Generation by Selectioncited by 40
  9. 2023Retrieval-Enhanced Contrastive Vision-Text Modelscited by 36
  10. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  11. 2025Analyzing and Boosting the Power of Fine-Grained Visual Recognition for Multi-modal Large Language Modelscited by 30
  12. 2024Advancing Social Intelligence in AI Agents: Technical Challenges and Open Questionscited by 21
20 more
  1. 2025I-Con: A Unifying Framework for Representation Learningcited by 15
  2. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Modelscited by 10
  3. 2024Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignmentcited by 8
  4. 2024Premonition: Using Generative Models to Preempt Future Data Changes in Continual Learningcited by 5
  5. 2024ChimpVLM: Ethogram-Enhanced Chimpanzee Behaviour Recognitioncited by 5
  6. 2024Tell, Don't Show!: Language Guidance Eases Transfer Across Domains in Images and Videoscited by 4
  7. 2025Tables Guide Vision: Learning to See the Heart through Tabular Datacited by 3
  8. 2025EcoWikiRS: Learning Ecological Representation of Satellite Images from Weak Supervision with Species Observations and Wikipediacited by 3
  9. 2023Compositionally Equivariant Representation Learningcited by 2
  10. 2025DiffCLIP: Differential Attention Meets CLIPcited by 2
  11. 2023MNN: Mixed Nearest-Neighbors for Self-Supervised Learningcited by 1
  12. 2024ConceptHash: Interpretable Fine-Grained Hashing via Concept Discoverycited by 1
  13. 2025CORA: Consistency-Guided Semi-Supervised Framework for Reasoning Segmentationcited by 1
  14. 2026Unifying Contrastive and Generative Objectives for Visual Understanding and Text-to-Image Generationcited by 1
  15. 2024Multi Positive Contrastive Learning with Pose-Consistent Generated Imagescited by 0
  16. 2025Impact of Language Guidance: A Reproducibility Studycited by 0
  17. 2025Amplifying Prominent Representations in Multimodal Learning via Variational Dirichlet Processcited by 0
  18. 2025Cross-Modal Alignment via Variational Copula Modellingcited by 0
  19. 2026Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Havecited by 0
  20. 2026Synesthesia via Direct Latent Augmentation:Bypassing the Decode-Encode Loop for Cross-Modal Distillationcited by 0