# SGRAM: Improving Scene Graph Parsing via Abstract Meaning Representation

arXiv: https://arxiv.org/abs/2210.08675 · HTML: https://mlsanity.com/paper/2210.08675/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 36 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 30 of 30)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [SPICE: Semantic Propositional Image Caption Evaluation](https://mlsanity.com/paper/1607.08822/) (2016)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [UNIMO: Towards Unified-Modal Understanding and Generation via Cross-Modal Contrastive Learning](https://mlsanity.com/paper/2012.15409/) (2020)
- [Investigating Pretrained Language Models for Graph-to-Text Generation](https://mlsanity.com/paper/2007.08426/) (2020)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [Semantic Image Manipulation Using Scene Graphs](https://mlsanity.com/paper/2004.03677/) (2020)
- [Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval](https://mlsanity.com/paper/1910.05134/) (2019)
- [Learning to Generate Scene Graph from Natural Language Supervision](https://mlsanity.com/paper/2109.02227/) (2021)
- [Neural AMR: Sequence-to-Sequence Models for Parsing and Generation](https://mlsanity.com/paper/1704.08381/) (2017)
- [Simple and Accurate Dependency Parsing Using Bidirectional LSTM Feature Representations](https://mlsanity.com/paper/1603.04351/) (2016)
- [Comprehensive Image Captioning via Scene Graph Decomposition](https://mlsanity.com/paper/2007.11731/) (2020)
- [AMR Parsing as Sequence-to-Graph Transduction](https://mlsanity.com/paper/1905.08704/) (2019)
- [AMR Parsing via Graph-Sequence Iterative Inference](https://mlsanity.com/paper/2004.05572/) (2020)
- [Modeling Graph Structure in Transformer for Better AMR-to-Text Generation](https://mlsanity.com/paper/1909.00136/) (2019)
- [Structured Query-Based Image Retrieval Using Scene Graphs](https://mlsanity.com/paper/2005.06653/) (2020)
- [In Defense of Scene Graphs for Image Captioning](https://mlsanity.com/paper/2102.04990/) (2021)
- [AMR Parsing with Action-Pointer Transformer](https://mlsanity.com/paper/2104.14674/) (2021)
- [Core Semantic First: A Top-down Approach for AMR Parsing](https://mlsanity.com/paper/1909.04303/) (2019)
- [Scene Graph Parsing as Dependency Parsing](https://mlsanity.com/paper/1803.09189/) (2018)
- [Understanding the Role of Scene Graphs in Visual Question Answering](https://mlsanity.com/paper/2101.05479/) (2021)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answering](https://mlsanity.com/paper/2107.06325/) (2021)
- [Incremental Object Grounding Using Scene Graphs](https://mlsanity.com/paper/2201.01901/) (2022)
- [Scene Graph Parsing by Attention Graph](https://mlsanity.com/paper/1909.06273/) (2019)

## Built on by (most cited first, 2 of 2)

- [CIC-BART-SSA: Controllable Image Captioning with Structured Semantic Augmentation](https://mlsanity.com/paper/2407.11393/) (2024)
- [All-in-One Conditioning for Text-to-Image Synthesis](https://mlsanity.com/paper/2602.09165/) (2026)

## Abstract (arXiv)

Scene graph is structured semantic representation that can be modeled as a form of graph from images and texts. Image-based scene graph generation research has been actively conducted until recently, whereas text-based scene graph generation research has not. In this paper, we focus on the problem of scene graph parsing from textual description of a visual scene. The core idea is to use abstract meaning representation (AMR) instead of the dependency parsing mainly used in previous studies. AMR is a graph-based semantic formalism of natural language which abstracts concepts of words in a sentence contrary to the dependency parsing which considers dependency relationships on all words in a sentence. To this end, we design a simple yet effective two-stage scene graph parsing framework utilizing abstract meaning representation, SGRAM (Scene GRaph parsing via Abstract Meaning representation): 1) transforming a textual description of an image into an AMR graph (Text-to-AMR) and 2) encoding the AMR graph into a Transformer-based language model to generate a scene graph (AMR-to-SG). Experimental results show the scene graphs generated by our framework outperforms the dependency parsing-based model by 11.61% and the previous state-of-the-art model using a pre-trained Transformer language model by 3.78%. Furthermore, we apply SGRAM to image retrieval task which is one of downstream tasks for scene graph, and confirm the effectiveness of scene graphs generated by our framework.