# Language-Conditioned Graph Networks for Relational Reasoning

arXiv: https://arxiv.org/abs/1905.04405 · HTML: https://mlsanity.com/paper/1905.04405/

## Key facts

- Cited by 55 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 May 2019.

## Built on by (most cited first, 30 of 55)

- [ViperGPT: Visual Inference via Python Execution for Reasoning](https://mlsanity.com/paper/2303.08128/) (2023)
- [STAR: A Benchmark for Situated Reasoning in Real-World Videos](https://mlsanity.com/paper/2405.09711/) (2024)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [Cross-Modal Progressive Comprehension for Referring Segmentation](https://mlsanity.com/paper/2105.07175/) (2021)
- [Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals](https://mlsanity.com/paper/2101.03737/) (2021)
- [Natural Language Understanding and Inference with MLLM in Visual Question Answering: A Survey](https://mlsanity.com/paper/2411.17558/) (2024)
- [Video as Conditional Graph Hierarchy for Multi-Granular Question Answering](https://mlsanity.com/paper/2112.06197/) (2021)
- [A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspective](https://mlsanity.com/paper/2209.13232/) (2022)
- [Neighbor-view Enhanced Model for Vision and Language Navigation](https://mlsanity.com/paper/2107.07201/) (2021)
- [Act As You Wish: Fine-Grained Control of Motion Diffusion Model with Hierarchical Semantic Graphs](https://mlsanity.com/paper/2311.01015/) (2023)
- [DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answering](https://mlsanity.com/paper/2107.04768/) (2021)
- [Greedy Gradient Ensemble for Robust Visual Question Answering](https://mlsanity.com/paper/2107.12651/) (2021)
- [PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoning](https://mlsanity.com/paper/2112.05136/) (2021)
- [eP-ALM: Efficient Perceptual Augmentation of Language Models](https://mlsanity.com/paper/2303.11403/) (2023)
- [Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications](https://mlsanity.com/paper/2302.00389/) (2023)
- [Context Disentangling and Prototype Inheriting for Robust Visual Grounding](https://mlsanity.com/paper/2312.11967/) (2023)
- [Found a Reason for me? Weakly-supervised Grounded Visual Question Answering using Capsules](https://mlsanity.com/paper/2105.04836/) (2021)
- [VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering](https://mlsanity.com/paper/2205.11501/) (2022)
- [Towards Lightweight Transformer via Group-wise Transformation for Vision-and-Language Tasks](https://mlsanity.com/paper/2204.07780/) (2022)
- [GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering](https://mlsanity.com/paper/2104.10283/) (2021)
- [Learning Situation Hyper-Graphs for Video Question Answering](https://mlsanity.com/paper/2304.08682/) (2023)
- [3D Concept Grounding on Neural Fields](https://mlsanity.com/paper/2207.06403/) (2022)
- [Towards Top-Down Reasoning: An Explainable Multi-Agent Approach for Visual Question Answering](https://mlsanity.com/paper/2311.17331/) (2023)
- [X-GGM: Graph Generative Modeling for Out-of-Distribution Generalization in Visual Question Answering](https://mlsanity.com/paper/2107.11576/) (2021)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [A Comprehensive Survey on Visual Question Answering Datasets and Algorithms](https://mlsanity.com/paper/2411.11150/) (2024)
- [General Greedy De-bias Learning](https://mlsanity.com/paper/2112.10572/) (2021)
- [Dynamic Key-value Memory Enhanced Multi-step Graph Reasoning for Knowledge-based Visual Question Answering](https://mlsanity.com/paper/2203.02985/) (2022)
- [Guiding Visual Question Answering with Attention Priors](https://mlsanity.com/paper/2205.12616/) (2022)
- [FlexCap: Describe Anything in Images in Controllable Detail](https://mlsanity.com/paper/2403.12026/) (2024)

## Abstract (arXiv)

Solving grounded language tasks often requires reasoning about relationships between objects in the context of a given task. For example, to answer the question "What color is the mug on the plate?" we must check the color of the specific mug that satisfies the "on" relationship with respect to the plate. Recent work has proposed various methods capable of complex relational reasoning. However, most of their power is in the inference structure, while the scene is represented with simple local appearance features. In this paper, we take an alternate approach and build contextualized representations for objects in a visual scene to support relational reasoning. We propose a general framework of Language-Conditioned Graph Networks (LCGN), where each node represents an object, and is described by a context-aware representation from related objects through iterative message passing conditioned on the textual input. E.g., conditioning on the "on" relationship to the plate, the object "mug" gathers messages from the object "plate" to update its representation to "mug on the plate", which can be easily consumed by a simple classifier for answer prediction. We experimentally show that our LCGN approach effectively supports relational reasoning and improves performance across several tasks and datasets. Our code is available at http://ronghanghu.com/lcgn.