# Composing Object Relations and Attributes for Image-Text Matching arXiv: https://arxiv.org/abs/2406.11820 ยท HTML: https://mlsanity.com/paper/2406.11820/ ## Key facts - Cited by 11 papers in the map (data through 14 Oct 2026). - Builds on 38 of its 58 references that are in the map. - First posted on arXiv 17 Jun 2024. ## Builds on (most cited first, 30 of 38) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [Aggregated Residual Transformations for Deep Neural Networks](https://mlsanity.com/paper/1611.05431/) (2016) - [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014) - [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019) - [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015) - [Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering](https://mlsanity.com/paper/1707.07998/) (2017) - [How Attentive are Graph Attention Networks?](https://mlsanity.com/paper/2105.14491/) (2021) - [Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2301.13826/) (2023) - [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021) - [Exploring the Limits of Weakly Supervised Pretraining](https://mlsanity.com/paper/1805.00932/) (2018) - [Stacked Cross Attention for Image-Text Matching](https://mlsanity.com/paper/1803.08024/) (2018) - [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017) - [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020) - [Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models](https://mlsanity.com/paper/1411.2539/) (2014) - [Visual Semantic Reasoning for Image-Text Matching](https://mlsanity.com/paper/1909.02701/) (2019) - [Grounded Text-to-Image Synthesis with Attention Refocusing](https://mlsanity.com/paper/2306.05427/) (2023) - [Probabilistic Embeddings for Cross-Modal Retrieval](https://mlsanity.com/paper/2101.05068/) (2021) - [Learning Deep Structure-Preserving Image-Text Embeddings](https://mlsanity.com/paper/1511.06078/) (2015) - [Learning the Best Pooling Strategy for Visual Semantic Embedding](https://mlsanity.com/paper/2011.04305/) (2020) - [Similarity Reasoning and Filtration for Image-Text Matching](https://mlsanity.com/paper/2101.01368/) (2021) - [Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks](https://mlsanity.com/paper/1909.00964/) (2019) - [Learning to Predict Visual Attributes in the Wild](https://mlsanity.com/paper/2106.09707/) (2021) - [Graph Structured Network for Image-Text Matching](https://mlsanity.com/paper/2004.00277/) (2020) - [Polysemous Visual-Semantic Embedding for Cross-Modal Retrieval](https://mlsanity.com/paper/1906.04402/) (2019) ## Built on by (most cited first, 11 of 11) - [CoLLM: A Large Language Model for Composed Image Retrieval](https://mlsanity.com/paper/2503.19910/) (2025) - [Universal Scene Graph Generation](https://mlsanity.com/paper/2503.15005/) (2025) - [Highlighting What Matters: Promptable Embeddings for Attribute-Focused Image Retrieval](https://mlsanity.com/paper/2505.15877/) (2025) - [GMM-Based Comprehensive Feature Extraction and Relative Distance Preservation For Few-Shot Cross-Modal Retrieval](https://mlsanity.com/paper/2505.13306/) (2025) - [ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval](https://mlsanity.com/paper/2602.01639/) (2026) - [Efficient and High-Fidelity Omni Modality Retrieval](https://mlsanity.com/paper/2603.02098/) (2026) - [V-Retrver: Evidence-Driven Agentic Reasoning for Universal Multimodal Retrieval](https://mlsanity.com/paper/2602.06034/) (2026) - [Ambiguity-Aware and High-Order Relation Learning for Multi-Grained Image-Text Matching](https://mlsanity.com/paper/2507.09256/) (2025) - [Visual Semantic Description Generation with MLLMs for Image-Text Matching](https://mlsanity.com/paper/2507.08590/) (2025) - [MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion](https://mlsanity.com/paper/2509.12901/) (2025) - [OS-HGAdapter: Open Semantic Hypergraph Adapter for Large Language Models Assisted Entropy-Enhanced Image-Text Alignment](https://mlsanity.com/paper/2510.13131/) (2025) ## Abstract (arXiv) We study the visual semantic embedding problem for image-text matching. Most existing work utilizes a tailored cross-attention mechanism to perform local alignment across the two image and text modalities. This is computationally expensive, even though it is more powerful than the unimodal dual-encoder approach. This work introduces a dual-encoder image-text matching model, leveraging a scene graph to represent captions with nodes for objects and attributes interconnected by relational edges. Utilizing a graph attention network, our model efficiently encodes object-attribute and object-object semantic relations, resulting in a robust and fast-performing system. Representing caption as a scene graph offers the ability to utilize the strong relational inductive bias of graph neural networks to learn object-attribute and object-object relations effectively. To train the model, we propose losses that align the image and caption both at the holistic level (image-caption) and the local level (image-object entity), which we show is key to the success of the model. Our model is termed Composition model for Object Relations and Attributes, CORA. Experimental results on two prominent image-text retrieval benchmarks, Flickr30K and MSCOCO, demonstrate that CORA outperforms existing state-of-the-art computationally expensive cross-attention methods regarding recall score while achieving fast computation speed of the dual encoder.