# HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation

arXiv: https://arxiv.org/abs/2403.12033 · HTML: https://mlsanity.com/paper/2403.12033/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 71 of its 94 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 71)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [ConceptNet 5.5: An Open Multilingual Graph of General Knowledge](https://mlsanity.com/paper/1612.03975/) (2016)
- [AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty](https://mlsanity.com/paper/1912.02781/) (2019)
- [Gated Graph Sequence Neural Networks](https://mlsanity.com/paper/1511.05493/) (2015)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [MEMO: Test Time Robustness via Adaptation and Augmentation](https://mlsanity.com/paper/2110.09506/) (2021)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [A Fourier Perspective on Model Robustness in Computer Vision](https://mlsanity.com/paper/1906.08988/) (2019)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Auto-Encoding Scene Graphs for Image Captioning](https://mlsanity.com/paper/1812.02378/) (2018)
- [Attend, Infer, Repeat: Fast Scene Understanding with Generative Models](https://mlsanity.com/paper/1603.08575/) (2016)
- [Learning 3D Semantic Scene Graphs from 3D Indoor Reconstructions](https://mlsanity.com/paper/2004.03967/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization](https://mlsanity.com/paper/2112.00463/) (2021)
- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Energy-Based Learning for Scene Graph Generation](https://mlsanity.com/paper/2103.02221/) (2021)
- [Scene Graph Generation with External Knowledge and Image Reconstruction](https://mlsanity.com/paper/1904.00560/) (2019)
- [Detecting Visual Relationships with Deep Relational Networks](https://mlsanity.com/paper/1704.03114/) (2017)

## Built on by (most cited first, 14 of 14)

- [Sigma: Siamese Mamba Network for Multi-Modal Semantic Segmentation](https://mlsanity.com/paper/2404.04256/) (2024)
- [Dual Prototype Evolving for Test-Time Generalization of Vision-Language Models](https://mlsanity.com/paper/2410.12790/) (2024)
- [Universal Scene Graph Generation](https://mlsanity.com/paper/2503.15005/) (2025)
- [LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation](https://mlsanity.com/paper/2411.00773/) (2024)
- [Learning Visual Conditioning Tokens to Correct Domain Shift for Fully Test-time Adaptation](https://mlsanity.com/paper/2406.19341/) (2024)
- [Bidirectional Prototype-Reward co-Evolution for Test-Time Adaptation of Vision-Language Models](https://mlsanity.com/paper/2503.09394/) (2025)
- [Dual-Path Adversarial Lifting for Domain Shift Correction in Online Test-time Adaptation](https://mlsanity.com/paper/2408.13983/) (2024)
- [Robo-SGG: Exploiting Layout-Oriented Normalization and Restitution Can Improve Robust Scene Graph Generation](https://mlsanity.com/paper/2504.12606/) (2025)
- [Hallucinate, Ground, Repeat: A Framework for Generalized Visual Relationship Detection](https://mlsanity.com/paper/2506.05651/) (2025)
- [THYME: Temporal Hierarchical-Cyclic Interactivity Modeling for Video Scene Graphs in Aerial Footage](https://mlsanity.com/paper/2507.09200/) (2025)
- [MSGFusion: Multimodal Scene Graph-Guided Infrared and Visible Image Fusion](https://mlsanity.com/paper/2509.12901/) (2025)
- [Generative Human-Object Interaction Detection via Differentiable Cognitive Steering of Multi-modal LLMs](https://mlsanity.com/paper/2512.17640/) (2025)
- [CURVE: Learning Causality-Inspired Invariant Representations for Robust Scene Understanding via Uncertainty-Guided Regularization](https://mlsanity.com/paper/2601.20355/) (2026)
- [Learning Context-Conditioned Predicate Semantics via Prototype Feedback](https://mlsanity.com/paper/2605.29610/) (2026)

## Abstract (arXiv)

Being able to understand visual scenes is a precursor for many downstream tasks, including autonomous driving, robotics, and other vision-based approaches. A common approach enabling the ability to reason over visual data is Scene Graph Generation (SGG); however, many existing approaches assume undisturbed vision, i.e., the absence of real-world corruptions such as fog, snow, smoke, as well as non-uniform perturbations like sun glare or water drops. In this work, we propose a novel SGG benchmark containing procedurally generated weather corruptions and other transformations over the Visual Genome dataset. Further, we introduce a corresponding approach, Hierarchical Knowledge Enhanced Robust Scene Graph Generation (HiKER-SGG), providing a strong baseline for scene graph generation under such challenging setting. At its core, HiKER-SGG utilizes a hierarchical knowledge graph in order to refine its predictions from coarse initial estimates to detailed predictions. In our extensive experiments, we show that HiKER-SGG does not only demonstrate superior performance on corrupted images in a zero-shot manner, but also outperforms current state-of-the-art methods on uncorrupted SGG tasks. Code is available at https://github.com/zhangce01/HiKER-SGG.