# VGSE: Visually-Grounded Semantic Embeddings for Zero-Shot Learning

arXiv: https://arxiv.org/abs/2203.10444 · HTML: https://mlsanity.com/paper/2203.10444/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 68 references that are in the map.
- First posted on arXiv 20 Mar 2022.

## Builds on (most cited first, 30 of 37)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1603.09246/) (2016)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [SCAN: Learning to Classify Images without Labels](https://mlsanity.com/paper/2005.12320/) (2020)
- [Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet](https://mlsanity.com/paper/1904.00760/) (2019)
- [Feature Generating Networks for Zero-Shot Learning](https://mlsanity.com/paper/1712.00981/) (2017)
- [Zero-Shot Learning Through Cross-Modal Transfer](https://mlsanity.com/paper/1301.3666/) (2013)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [FSRNet: End-to-End Learning Face Super-Resolution with Facial Priors](https://mlsanity.com/paper/1711.10703/) (2017)
- [Evaluation of Output Embeddings for Fine-Grained Image Classification](https://mlsanity.com/paper/1409.8403/) (2014)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [f-VAEGAN-D2: A Feature Generating Framework for Any-Shot Learning](https://mlsanity.com/paper/1903.10132/) (2019)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [Zero-shot Recognition via Semantic Embeddings and Knowledge Graphs](https://mlsanity.com/paper/1803.08035/) (2018)
- [Latent Embeddings for Zero-shot Classification](https://mlsanity.com/paper/1603.08895/) (2016)
- [A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts](https://mlsanity.com/paper/1712.01381/) (2017)
- [Rethinking Knowledge Graph Propagation for Zero-Shot Learning](https://mlsanity.com/paper/1805.11724/) (2018)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)
- [Semantic-Guided Multi-Attention Localization for Zero-Shot Learning](https://mlsanity.com/paper/1903.00502/) (2019)
- [Less is more: zero-shot learning from online textual documents with noise suppression](https://mlsanity.com/paper/1604.01146/) (2016)
- [Generating Visual Representations for Zero-Shot Classification](https://mlsanity.com/paper/1708.06975/) (2017)
- [Fashion Forward: Forecasting Visual Style in Fashion](https://mlsanity.com/paper/1705.06394/) (2017)
- [Recovering the Missing Link: Predicting Class-Attribute Associations for Unsupervised Zero-Shot Learning](https://mlsanity.com/paper/1610.04787/) (2016)
- [Learning the Latent "Look": Unsupervised Discovery of a Style-Coherent Embedding from Fashion Images](https://mlsanity.com/paper/1707.03376/) (2017)

## Built on by (most cited first, 27 of 27)

- [I2MVFormer: Large Language Model Generated Multi-View Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2212.02291/) (2022)
- [I2DFormer: Learning Image to Document Attention for Zero-Shot Image Classification](https://mlsanity.com/paper/2209.10304/) (2022)
- [Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning](https://mlsanity.com/paper/2404.14808/) (2024)
- [Image-free Classifier Injection for Zero-Shot Classification](https://mlsanity.com/paper/2308.10599/) (2023)
- [Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects](https://mlsanity.com/paper/2401.17766/) (2024)
- [Learning Adversarial Semantic Embeddings for Zero-Shot Recognition in Open Worlds](https://mlsanity.com/paper/2307.03416/) (2023)
- [Training-Free Personalization via Retrieval and Reasoning on Fingerprints](https://mlsanity.com/paper/2503.18623/) (2025)
- [Rebalanced Zero-shot Learning](https://mlsanity.com/paper/2210.07031/) (2022)
- [Holistic Transfer: Towards Non-Disruptive Fine-Tuning with Partial Target Data](https://mlsanity.com/paper/2311.01420/) (2023)
- [Attribute-Aware Representation Rectification for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2311.14750/) (2023)
- [AlignZeg: Mitigating Objective Misalignment for Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2404.05667/) (2024)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [ParsNets: A Parsimonious Orthogonal and Low-Rank Linear Networks for Zero-Shot Learning](https://mlsanity.com/paper/2312.09709/) (2023)
- [ProKD: An Unsupervised Prototypical Knowledge Distillation Network for Zero-Resource Cross-Lingual Named Entity Recognition](https://mlsanity.com/paper/2301.08855/) (2023)
- [MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2503.06847/) (2025)
- [Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation](https://mlsanity.com/paper/2306.08487/) (2023)
- [`Eyes of a Hawk and Ears of a Fox': Part Prototype Network for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2404.08761/) (2024)
- [Interpretable Zero-shot Learning with Infinite Class Concepts](https://mlsanity.com/paper/2505.03361/) (2025)
- [Improving Zero-Shot Action Recognition using Human Instruction with Text Description](https://mlsanity.com/paper/2301.08874/) (2023)
- [Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2306.01929/) (2023)
- [Attribute Localization and Revision Network for Zero-Shot Learning](https://mlsanity.com/paper/2310.07548/) (2023)
- [SEER-ZSL: Semantic Encoder-Enhanced Representations for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2312.13100/) (2023)
- [Bridge the Gap Between Visual and Linguistic Comprehension for Generalized Zero-shot Semantic Segmentation](https://mlsanity.com/paper/2503.23806/) (2025)
- [Prototype-Guided Curriculum Learning for Zero-Shot Learning](https://mlsanity.com/paper/2508.07771/) (2025)
- [Adversarial Robustness in Zero-Shot Learning:An Empirical Study on Class and Concept-Level Vulnerabilities](https://mlsanity.com/paper/2512.18651/) (2025)
- [Incentivizing Generative Zero-Shot Learning via Outcome-Reward Reinforcement Learning with Visual Cues](https://mlsanity.com/paper/2603.21138/) (2026)
- [CAST: Closed-form Analytic Semantic Transfer for Zero-Shot Classifier Extension](https://mlsanity.com/paper/2608.13751/) (2026)

## Abstract (arXiv)

Human-annotated attributes serve as powerful semantic embeddings in zero-shot learning. However, their annotation process is labor-intensive and needs expert supervision. Current unsupervised semantic embeddings, i.e., word embeddings, enable knowledge transfer between classes. However, word embeddings do not always reflect visual similarities and result in inferior zero-shot performance. We propose to discover semantic embeddings containing discriminative visual properties for zero-shot learning, without requiring any human annotation. Our model visually divides a set of images from seen classes into clusters of local image regions according to their visual similarity, and further imposes their class discrimination and semantic relatedness. To associate these clusters with previously unseen classes, we use external knowledge, e.g., word embeddings and propose a novel class relation discovery module. Through quantitative and qualitative evaluation, we demonstrate that our model discovers semantic embeddings that model the visual properties of both seen and unseen classes. Furthermore, we demonstrate on three benchmarks that our visually-grounded semantic embeddings further improve performance over word embeddings across various ZSL models by a large margin.