Data to 7 Oct 2026

Visual Relationship Detection with Language Priors

arXiv 1608.00187 ↗First posted 31 Jul 2016cs.CVcited by 333

Not on the map. Why?

Abstract (arXiv)

Visual relationships capture a wide variety of interactions between pairs of objects in images (e.g. "man riding bicycle" and "man pushing bicycle"). Consequently, the set of possible relationships is extremely large and it is difficult to obtain sufficient training examples for all possible relationships. Because of this limitation, previous work on visual relationship detection has concentrated on predicting only a handful of relationships. Though most relationships are infrequent, their objects (e.g. "man" and "bicycle") and predicates (e.g. "riding" and "pushing") independently occur more frequently. We propose a model that uses this insight to train visual models for objects and predicates individually and later combines them together to predict multiple relationships per image. We improve on prior work by leveraging language priors from semantic word embeddings to finetune the likelihood of a predicted relationship. Our model can scale to predict thousands of types of relationships from a few examples. Additionally, we localize the objects in the predicted relationships as bounding boxes in the image. We further demonstrate that understanding relationships can improve content based image retrieval.

Key facts. Cited by 333 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Jul 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 0; 2021: 68; 2022: 71; 2023: 71; 2024: 61; 2025: 33; 2026: 28.

333 papers.

Sort:

By area:

2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
2021ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmeticcited by 150
2021Kimera: from SLAM to Spatial Perception with 3D Dynamic Scene Graphscited by 144
2021OW-DETR: Open-world Detection Transformercited by 129
2021A Comprehensive Survey of Scene Graphs: Generation and Applicationcited by 125
2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
2022Panoptic Scene Graph Generationcited by 114
2021Learning Graph Embeddings for Compositional Zero-shot Learningcited by 112

38 more

2021Probing Image-Language Transformers for Verb Understandingcited by 109
2021Energy-Based Learning for Scene Graph Generationcited by 104
2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
2022RelTR: Relation Transformer for Scene Graph Generationcited by 95
2024The All-Seeing Project V2: Towards General Relation Comprehension of the Open Worldcited by 90
2022CLIP-Event: Connecting Text and Images with Event Structurescited by 86
2023OpenLane-V2: A Topology Reasoning Benchmark for Unified 3D HD Mappingcited by 83
2023ReVersion: Diffusion-Based Relation Inversion from Imagescited by 82
2021Detecting Human-Object Interaction via Fabricated Compositional Learningcited by 79
2024Open3DSG: Open-Vocabulary 3D Scene Graphs from Point Clouds with Queryable Objects and Open-Set Relationshipscited by 76
2022Multi-Modal Knowledge Graph Construction and Application: A Surveycited by 74
2022Scene Graph Generation: A Comprehensive Surveycited by 73
2021Recovering the Unbiased Scene Graphs from the Biased Onescited by 71
2024Perception Tokens Enhance Visual Reasoning in Multimodal Language Modelscited by 68
2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
2022Stacked Hybrid-Attention and Group Collaborative Learning for Unbiased Scene Graph Generationcited by 67
2022Relationformer: A Unified Framework for Image-to-Graph Generationcited by 67
2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
2022The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generationcited by 60
2022Fine-Grained Scene Graph Generation with Data Transfercited by 57
2023DreamSync: Aligning Text-to-Image Generation with Image Understanding Feedbackcited by 57
2022Disentangling Visual Embeddings for Attributes and Objectscited by 55
2022Towards Open-vocabulary Scene Graph Generation with Prompt-based Finetuningcited by 54
2025Open-Vocabulary Functional 3D Scene Graphs for Real-World Indoor Spacescited by 53
2021Target Adaptive Context Aggregation for Video Scene Graph Generationcited by 51
2021From General to Specific: Informative Scene Graph Generation via Balance Adjustmentcited by 51
2021Learning Graph Embeddings for Open World Compositional Zero-Shot Learningcited by 49
2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
2021Exploiting Edge-Oriented Reasoning for 3D Point-based Scene Graph Analysiscited by 47
2021A Review on Explainability in Multimodal Deep Neural Netscited by 46
2024EGTR: Extracting Graph from Transformer for Scene Graph Generationcited by 45
2021Structured Sparse R-CNN for Direct Scene Graph Generationcited by 44
2024Revisiting Text-to-Image Evaluation with Gecko: On Metrics, Prompts, and Human Ratingscited by 44
2023VL-SAT: Visual-Linguistic Semantics Assisted Training for 3D Semantic Scene Graph Prediction in Point Cloudcited by 42
2022UC-OWOD: Unknown-Classified Open World Object Detectioncited by 40
2022Continuous Scene Representations for Embodied AIcited by 39
2023Incremental 3D Semantic Scene Graph Prediction from RGB Sequencescited by 39
2021Unified Graph Structured Models for Video Understandingcited by 38

50 most cited of 333 All 333 (JSON)