# Scene Graph Parsing as Dependency Parsing

arXiv: https://arxiv.org/abs/1803.09189 · HTML: https://mlsanity.com/paper/1803.09189/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2018.

## Built on by (most cited first, 21 of 21)

- [Self-Supervised Multimodal Learning: A Survey](https://mlsanity.com/paper/2304.01008/) (2023)
- [FACTUAL: A Benchmark for Faithful and Consistent Textual Scene Graph Parsing](https://mlsanity.com/paper/2305.17497/) (2023)
- [Deep Learning Approaches on Image Captioning: A Review](https://mlsanity.com/paper/2201.12944/) (2022)
- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [Relation-aware Instance Refinement for Weakly Supervised Visual Grounding](https://mlsanity.com/paper/2103.12989/) (2021)
- [Linguistic Structures as Weak Supervision for Visual Scene Graph Generation](https://mlsanity.com/paper/2105.13994/) (2021)
- [Scene Graph as Pivoting: Inference-time Image-free Unsupervised Multimodal Machine Translation with Visual Scene Hallucination](https://mlsanity.com/paper/2305.12256/) (2023)
- [Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling](https://mlsanity.com/paper/2305.11719/) (2023)
- [Cross2StrA: Unpaired Cross-lingual Image Captioning with Cross-lingual Cross-modal Structure-pivoted Alignment](https://mlsanity.com/paper/2305.12260/) (2023)
- [Visually Grounded Concept Composition](https://mlsanity.com/paper/2109.14115/) (2021)
- [Differentiable Parsing and Visual Grounding of Natural Language Instructions for Object Placement](https://mlsanity.com/paper/2210.00215/) (2022)
- [Weakly-Supervised 3D Scene Graph Generation via Visual-Linguistic Assisted Pseudo-labeling](https://mlsanity.com/paper/2404.02527/) (2024)
- [DescribeEarth: Describe Anything for Remote Sensing Images](https://mlsanity.com/paper/2509.25654/) (2025)
- [Scene Graph Modification as Incremental Structure Expanding](https://mlsanity.com/paper/2209.09093/) (2022)
- [SGRAM: Improving Scene Graph Parsing via Abstract Meaning Representation](https://mlsanity.com/paper/2210.08675/) (2022)
- [Grounding Scene Graphs on Natural Images via Visio-Lingual Message Passing](https://mlsanity.com/paper/2211.01969/) (2022)
- [Multimodal Machine Translation with Visual Scene Graph Pruning](https://mlsanity.com/paper/2505.19507/) (2025)
- [DiscoSG: Towards Discourse-Level Text Scene Graph Parsing through Iterative Graph Refinement](https://mlsanity.com/paper/2506.15583/) (2025)
- [GIIFT: Graph-guided Inductive Image-free Multimodal Machine Translation](https://mlsanity.com/paper/2507.18562/) (2025)
- [Semi-Supervised Image Captioning Considering Wasserstein Graph Matching](https://mlsanity.com/paper/2403.17995/) (2024)
- [Modeling Scientific Experiment Scenes: Dataset and Model](https://mlsanity.com/paper/2608.02892/) (2026)

## Abstract (arXiv)

In this paper, we study the problem of parsing structured knowledge graphs from textual descriptions. In particular, we consider the scene graph representation that considers objects together with their attributes and relations: this representation has been proved useful across a variety of vision and language applications. We begin by introducing an alternative but equivalent edge-centric view of scene graphs that connect to dependency parses. Together with a careful redesign of label and action space, we combine the two-stage pipeline used in prior work (generic dependency parsing followed by simple post-processing) into one, enabling end-to-end training. The scene graphs generated by our learned neural dependency parser achieve an F-score similarity of 49.67% to ground truth graphs on our evaluation set, surpassing best previous approaches by 5%. We further demonstrate the effectiveness of our learned parser on image retrieval applications.