# Conditional Image-Text Embedding Networks

arXiv: https://arxiv.org/abs/1711.08389 · HTML: https://mlsanity.com/paper/1711.08389/

## Key facts

- Cited by 50 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Nov 2017.

## Built on by (most cited first, 30 of 50)

- [MDETR -- Modulated Detection for End-to-End Multi-Modal Understanding](https://mlsanity.com/paper/2104.12763/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Referring Transformer: A One-step Approach to Multi-task Visual Grounding](https://mlsanity.com/paper/2106.03089/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Improving Visual Grounding with Visual-Linguistic Verification and Iterative Reasoning](https://mlsanity.com/paper/2205.00272/) (2022)
- [Look Before You Leap: Learning Landmark Features for One-Stage Visual Grounding](https://mlsanity.com/paper/2104.04386/) (2021)
- [Toward Visual Grounding: A Survey](https://mlsanity.com/paper/2412.20206/) (2024)
- [Visual Grounding with Transformers](https://mlsanity.com/paper/2105.04281/) (2021)
- [TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer](https://mlsanity.com/paper/2206.06619/) (2022)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [SceneTrilogy: On Human Scene-Sketch and its Complementarity with Photo and Text](https://mlsanity.com/paper/2204.11964/) (2022)
- [Dynamic MDETR: A Dynamic Multimodal Transformer Decoder for Visual Grounding](https://mlsanity.com/paper/2209.13959/) (2022)
- [DQ-DETR: Dual Query Detection Transformer for Phrase Extraction and Grounding](https://mlsanity.com/paper/2211.15516/) (2022)
- [Context Disentangling and Prototype Inheriting for Robust Visual Grounding](https://mlsanity.com/paper/2312.11967/) (2023)
- [Adapting CLIP For Phrase Localization Without Further Training](https://mlsanity.com/paper/2204.03647/) (2022)
- [Iterative Shrinking for Referring Expression Grounding Using Deep Reinforcement Learning](https://mlsanity.com/paper/2103.05187/) (2021)
- [Ground then Navigate: Language-guided Navigation in Dynamic Scenes](https://mlsanity.com/paper/2209.11972/) (2022)
- [Disentangled Motif-aware Graph Learning for Phrase Grounding](https://mlsanity.com/paper/2104.06008/) (2021)
- [YORO -- Lightweight End to End Visual Grounding](https://mlsanity.com/paper/2211.07912/) (2022)
- [Deconfounded Visual Grounding](https://mlsanity.com/paper/2112.15324/) (2021)
- [Toward Explainable and Fine-Grained 3D Grounding through Referring Textual Phrases](https://mlsanity.com/paper/2207.01821/) (2022)
- [SegVG: Transferring Object Bounding Box to Segmentation for Visual Grounding](https://mlsanity.com/paper/2407.03200/) (2024)
- [DSE-GAN: Dynamic Semantic Evolution Generative Adversarial Network for Text-to-Image Generation](https://mlsanity.com/paper/2209.01339/) (2022)
- [Uncertainty-aware Medical Diagnostic Phrase Identification and Grounding](https://mlsanity.com/paper/2404.06798/) (2024)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval](https://mlsanity.com/paper/2403.07222/) (2024)
- [Correspondence Matters for Video Referring Expression Comprehension](https://mlsanity.com/paper/2207.10400/) (2022)
- [Comprehensive Multi-Modal Interactions for Referring Image Segmentation](https://mlsanity.com/paper/2104.10412/) (2021)
- [Effectively Leveraging Attributes for Visual Similarity](https://mlsanity.com/paper/2105.01695/) (2021)

## Abstract (arXiv)

This paper presents an approach for grounding phrases in images which jointly learns multiple text-conditioned embeddings in a single end-to-end model. In order to differentiate text phrases into semantically distinct subspaces, we propose a concept weight branch that automatically assigns phrases to embeddings, whereas prior works predefine such assignments. Our proposed solution simplifies the representation requirements for individual embeddings and allows the underrepresented concepts to take advantage of the shared representations before feeding them into concept-specific layers. Comprehensive experiments verify the effectiveness of our approach across three phrase grounding datasets, Flickr30K Entities, ReferIt Game, and Visual Genome, where we obtain a (resp.) 4%, 3%, and 4% improvement in grounding performance over a strong region-phrase embedding baseline.