# DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning

arXiv: https://arxiv.org/abs/2207.01328 · HTML: https://mlsanity.com/paper/2207.01328/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 78 references that are in the map.
- First posted on arXiv 4 Jul 2022.

## Builds on (most cited first, 30 of 49)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [LXMERT: Learning Cross-Modality Encoder Representations from Transformers](https://mlsanity.com/paper/1908.07490/) (2019)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Multi-Task Learning as Multi-Objective Optimization](https://mlsanity.com/paper/1810.04650/) (2018)
- [Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Ugly](https://mlsanity.com/paper/1707.00600/) (2017)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Label-Embedding for Image Classification](https://mlsanity.com/paper/1503.08677/) (2015)
- [Attribute Prototype Network for Zero-Shot Learning](https://mlsanity.com/paper/2008.08290/) (2020)
- [A Review of Generalized Zero-Shot Learning Methods](https://mlsanity.com/paper/2011.08641/) (2020)
- [Semantic Autoencoder for Zero-Shot Learning](https://mlsanity.com/paper/1704.08345/) (2017)
- [Synthesized Classifiers for Zero-Shot Learning](https://mlsanity.com/paper/1603.00550/) (2016)
- [Learning Graph Embeddings for Compositional Zero-shot Learning](https://mlsanity.com/paper/2102.01987/) (2021)
- [A Generative Adversarial Approach for Zero-Shot Learning from Noisy Texts](https://mlsanity.com/paper/1712.01381/) (2017)
- [Counterfactual Zero-Shot and Open-Set Visual Recognition](https://mlsanity.com/paper/2103.00887/) (2021)
- [Contrastive Embedding for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2103.16173/) (2021)
- [Rethinking Knowledge Graph Propagation for Zero-Shot Learning](https://mlsanity.com/paper/1805.11724/) (2018)
- [Domain-aware Visual Bias Eliminating for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2003.13261/) (2020)
- [Goal-Oriented Gaze Estimation for Zero-Shot Learning](https://mlsanity.com/paper/2103.03433/) (2021)

## Built on by (most cited first, 30 of 33)

- [Making Large Language Models Perform Better in Knowledge Graph Completion](https://mlsanity.com/paper/2310.06671/) (2023)
- [Decomposed Soft Prompt Guided Fusion Enhancing for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2211.10681/) (2022)
- [Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations](https://mlsanity.com/paper/2305.06152/) (2023)
- [Progressive Semantic-Visual Mutual Adaption for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2303.15322/) (2023)
- [Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment](https://mlsanity.com/paper/2307.16210/) (2023)
- [Progressive Semantic-Guided Vision Transformer for Zero-Shot Learning](https://mlsanity.com/paper/2404.07713/) (2024)
- [Visual-Augmented Dynamic Semantic Prototype for Generative Zero-Shot Learning](https://mlsanity.com/paper/2404.14808/) (2024)
- [Fine-Grained Zero-Shot Learning: Advances, Challenges, and Prospects](https://mlsanity.com/paper/2401.17766/) (2024)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [ZeroDDI: A Zero-Shot Drug-Drug Interaction Event Prediction Method with Semantic Enhanced Learning and Dual-Modal Uniform Alignment](https://mlsanity.com/paper/2407.00891/) (2024)
- [On the Element-Wise Representation and Reasoning in Zero-Shot Image Recognition: A Systematic Survey](https://mlsanity.com/paper/2408.04879/) (2024)
- [GenZSL: Generative Zero-Shot Learning Via Inductive Variational Autoencoder](https://mlsanity.com/paper/2505.11882/) (2025)
- [Instances and Labels: Hierarchy-aware Joint Supervised Contrastive Learning for Hierarchical Multi-Label Text Classification](https://mlsanity.com/paper/2310.05128/) (2023)
- [Enhancing Visual Continual Learning with Language-Guided Supervision](https://mlsanity.com/paper/2403.16124/) (2024)
- [Visual-Semantic Decomposition and Partial Alignment for Document-based Zero-Shot Learning](https://mlsanity.com/paper/2407.15613/) (2024)
- [Visual and Semantic Prompt Collaboration for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2503.23030/) (2025)
- [HOPE: A Memory-Based and Composition-Aware Framework for Zero-Shot Learning with Hopfield Network and Soft Mixture of Experts](https://mlsanity.com/paper/2311.14747/) (2023)
- [PSVMA+: Exploring Multi-granularity Semantic-visual Adaption for Generalized Zero-shot Learning](https://mlsanity.com/paper/2410.11560/) (2024)
- [SkeletonContext: Skeleton-side Context Prompt Learning for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2603.29692/) (2026)
- [Diverse and Tailored Image Generation for Zero-shot Multi-label Classification](https://mlsanity.com/paper/2404.03144/) (2024)
- [MADS: Multi-Attribute Document Supervision for Zero-Shot Image Classification](https://mlsanity.com/paper/2503.06847/) (2025)
- [A Novel Neural-symbolic System under Statistical Relational Learning](https://mlsanity.com/paper/2309.08931/) (2023)
- [Semantics-enhanced Cross-modal Masked Image Modeling for Vision-Language Pre-training](https://mlsanity.com/paper/2403.00249/) (2024)
- [Interpretable Zero-shot Learning with Infinite Class Concepts](https://mlsanity.com/paper/2505.03361/) (2025)
- [Recent Advances of Local Mechanisms in Computer Vision: A Survey and Outlook of Recent Work](https://mlsanity.com/paper/2306.01929/) (2023)
- [SEER-ZSL: Semantic Encoder-Enhanced Representations for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2312.13100/) (2023)
- [Cross-Modal Coordination Across a Diverse Set of Input Modalities](https://mlsanity.com/paper/2401.16347/) (2024)
- [Few-Shot Inspired Generative Zero-Shot Learning](https://mlsanity.com/paper/2507.01026/) (2025)
- [Prototype-Guided Curriculum Learning for Zero-Shot Learning](https://mlsanity.com/paper/2508.07771/) (2025)
- [Multi-Granularity Mutual Refinement Network for Zero-Shot Learning](https://mlsanity.com/paper/2511.08163/) (2025)

## Abstract (arXiv)

Zero-shot learning (ZSL) aims to predict unseen classes whose samples have never appeared during training. One of the most effective and widely used semantic information for zero-shot image classification are attributes which are annotations for class-level visual characteristics. However, the current methods often fail to discriminate those subtle visual distinctions between images due to not only the shortage of fine-grained annotations, but also the attribute imbalance and co-occurrence. In this paper, we present a transformer-based end-to-end ZSL method named DUET, which integrates latent semantic knowledge from the pre-trained language models (PLMs) via a self-supervised multi-modal learning paradigm. Specifically, we (1) developed a cross-modal semantic grounding network to investigate the model's capability of disentangling semantic attributes from the images; (2) applied an attribute-level contrastive learning strategy to further enhance the model's discrimination on fine-grained visual characteristics against the attribute co-occurrence and imbalance; (3) proposed a multi-task learning policy for considering multi-model objectives. We find that our DUET can achieve state-of-the-art performance on three standard ZSL benchmarks and a knowledge graph equipped ZSL benchmark. Its components are effective and its predictions are interpretable.