mlsanityData to 7 Oct 2026

Semantically Grounded Visual Embeddings for Zero-Shot Learning

arXiv 2201.00577 ↗First posted 3 Jan 2022cs.CVcited by 4

Not on the map. Why?

Abstract (arXiv)

Zero-shot learning methods rely on fixed visual and semantic embeddings, extracted from independent vision and language models, both pre-trained for other large-scale tasks. This is a weakness of current zero-shot learning frameworks as such disjoint embeddings fail to adequately associate visual and textual information to their shared semantic content. Therefore, we propose to learn semantically grounded and enriched visual information by computing a joint image and text model with a two-stream network on a proxy task. To improve this alignment between image and textual representations, provided by attributes, we leverage ancillary captions to provide grounded semantic information. Our method, dubbed joint embeddings for zero-shot learning is evaluated on several benchmark datasets, improving the performance of existing state-of-the-art methods in both standard (+1.6% on aPY, +2.6% on FLO) and generalized (+2.1% on AWA2, +2.2% on CUB) zero-shot recognition.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 36 of its 57 references that are in the map. First posted on arXiv 3 Jan 2022.

Builds on

36 of 57 references.

References · 36

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  7. 2015VQA: Visual Question Answeringcited by 2,489
  8. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
24 more
  1. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  2. 2017Feature Generating Networks for Zero-Shot Learningcited by 257
  3. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  4. 2016Learning Deep Representations of Fine-grained Visual Descriptionscited by 182
  5. 2014Evaluation of Output Embeddings for Fine-Grained Image Classificationcited by 166
  6. 2019f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learningcited by 164
  7. 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
  8. 2020ImageBERT: Cross-modal Pre-training with Large-scale Weak-supervised Image-Text Datacited by 136
  9. 2017Semantic Autoencoder for Zero-Shot Learningcited by 133
  10. 2016Latent Embeddings for Zero-shot Classificationcited by 129
  11. 2016Synthesized Classifiers for Zero-Shot Learningcited by 113
  12. 2018Multi-modal Cycle-consistent Generalized Zero-Shot Learningcited by 87
  13. 2019Episode-based Prototype Generating Network for Zero-Shot Learningcited by 66
  14. 2019Transferable Contrastive Network for Generalized Zero-Shot Learningcited by 64
  15. 2018Generative Dual Adversarial Network for Generalized Zero-shot Learningcited by 61
  16. 2018Transductive Unbiased Embedding for Zero-Shot Learningcited by 44
  17. 2017A Simple Exponential Family Framework for Zero-Shot Learningcited by 38
  18. 2019Learning Feature-to-Feature Translator by Alternating Back-Propagation for Generative Zero-Shot Learningcited by 38
  19. 2020Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognitioncited by 34
  20. 2019Aiding Intra-Text Representations with Visual Context for Multimodal Named Entity Recognitioncited by 15
  21. 2018Learning Class Prototypes via Structure Alignment for Zero-Shot Recognitioncited by 14
  22. 2018Zero-Shot Kernel Learningcited by 11
  23. 2018Classifier and Exemplar Synthesis for Zero-Shot Learningcited by 9
  24. 2019Do Cross Modal Systems Leverage Semantic Relationships?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 1; 2024: 1; 2025: 0; 2026: 1.

4 papers.

  1. 2022DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learningcited by 33
  2. 2023Single-branch Network for Multimodal Trainingcited by 13
  3. 2024ZeroDDI: A Zero-Shot Drug-Drug Interaction Event Prediction Method with Semantic Enhanced Learning and Dual-Modal Uniform Alignmentcited by 5
  4. 2026SMETA-ZSL:Semantic Meta-Alignment for Zero-Shot Threat Classificationcited by 0