# The Devil is in the Labels: Noisy Label Correction for Robust Scene Graph Generation

arXiv: https://arxiv.org/abs/2206.03014 · HTML: https://mlsanity.com/paper/2206.03014/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 44 references that are in the map.
- First posted on arXiv 7 Jun 2022.

## Builds on (most cited first, 30 of 39)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels](https://mlsanity.com/paper/1805.07836/) (2018)
- [Large-Scale Long-Tailed Recognition in an Open World](https://mlsanity.com/paper/1904.05160/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Symmetric Cross Entropy for Robust Learning with Noisy Labels](https://mlsanity.com/paper/1908.06112/) (2019)
- [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017)
- [Neural Motifs: Scene Graph Parsing with Global Context](https://mlsanity.com/paper/1711.06640/) (2017)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016)
- [Graph R-CNN for Scene Graph Generation](https://mlsanity.com/paper/1808.00191/) (2018)
- [Learning to Compose Dynamic Tree Structures for Visual Contexts](https://mlsanity.com/paper/1812.01880/) (2018)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [CleanNet: Transfer Learning for Scalable Image Classifier Training with Label Noise](https://mlsanity.com/paper/1711.07131/) (2017)
- [Dimensionality-Driven Learning with Noisy Labels](https://mlsanity.com/paper/1806.02612/) (2018)
- [GPS-Net: Graph Property Sensing Network for Scene Graph Generation](https://mlsanity.com/paper/2003.12962/) (2020)
- [Knowledge-Embedded Routing Network for Scene Graph Generation](https://mlsanity.com/paper/1903.03326/) (2019)
- [Learning from Noisy Labels with Distillation](https://mlsanity.com/paper/1703.02391/) (2017)
- [Scene Graph Generation from Objects, Phrases and Region Captions](https://mlsanity.com/paper/1707.09700/) (2017)
- [CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention](https://mlsanity.com/paper/2108.00154/) (2021)
- [Bipartite Graph Network with Adaptive Message Passing for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2104.00308/) (2021)
- [CurriculumNet: Weakly Supervised Learning from Large-Scale Web Images](https://mlsanity.com/paper/1808.01097/) (2018)
- [Visual Translation Embedding Network for Visual Relation Detection](https://mlsanity.com/paper/1702.08319/) (2017)
- [Toward Robustness against Label Noise in Training Deep Discriminative Neural Networks](https://mlsanity.com/paper/1706.00038/) (2017)
- [PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2009.00893/) (2020)

## Built on by (most cited first, 30 of 60)

- [Panoptic Scene Graph Generation](https://mlsanity.com/paper/2207.11247/) (2022)
- [Panoptic Video Scene Graph Generation](https://mlsanity.com/paper/2311.17058/) (2023)
- [From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Models](https://mlsanity.com/paper/2404.00906/) (2024)
- [Prototype-based Embedding Network for Scene Graph Generation](https://mlsanity.com/paper/2303.07096/) (2023)
- [Visually-Prompted Language Model for Fine-Grained Scene Graph Generation in an Open World](https://mlsanity.com/paper/2303.13233/) (2023)
- [Zero-shot Visual Relation Detection via Composite Visual Cues from Large Language Models](https://mlsanity.com/paper/2305.12476/) (2023)
- [Compositional Prompt Tuning with Motion Cues for Open-vocabulary Video Relation Detection](https://mlsanity.com/paper/2302.00268/) (2023)
- [HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understanding](https://mlsanity.com/paper/2312.03050/) (2023)
- [Integrating Object-aware and Interaction-aware Knowledge for Weakly Supervised Scene Graph Generation](https://mlsanity.com/paper/2208.01834/) (2022)
- [NICEST: Noisy Label Correction and Training for Robust Scene Graph Generation](https://mlsanity.com/paper/2207.13316/) (2022)
- [HiLo: Exploiting High Low Frequency Relations for Unbiased Panoptic Scene Graph Generation](https://mlsanity.com/paper/2303.15994/) (2023)
- [Unbiased Heterogeneous Scene Graph Generation with Relation-aware Message Passing Neural Network](https://mlsanity.com/paper/2212.00443/) (2022)
- [Scene Graph Generation with Role-Playing Large Language Models](https://mlsanity.com/paper/2410.15364/) (2024)
- [Compositional Feature Augmentation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2308.06712/) (2023)
- [Adaptive Fine-Grained Predicates Learning for Scene Graph Generation](https://mlsanity.com/paper/2207.04602/) (2022)
- [Label Semantic Knowledge Distillation for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.03763/) (2022)
- [HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation](https://mlsanity.com/paper/2403.12033/) (2024)
- [Neural Clustering based Visual Representation Learning](https://mlsanity.com/paper/2403.17409/) (2024)
- [Adaptive Self-training Framework for Fine-grained Scene Graph Generation](https://mlsanity.com/paper/2401.09786/) (2024)
- [DSGG: Dense Relation Transformer for an End-to-end Scene Graph Generation](https://mlsanity.com/paper/2403.14886/) (2024)
- [Groupwise Query Specialization and Quality-Aware Multi-Assignment for Transformer-based Visual Relationship Detection](https://mlsanity.com/paper/2403.17709/) (2024)
- [DIFFVSGG: Diffusion-Driven Online Video Scene Graph Generation](https://mlsanity.com/paper/2503.13957/) (2025)
- [Dual-branch Hybrid Learning Network for Unbiased Scene Graph Generation](https://mlsanity.com/paper/2207.07913/) (2022)
- [Hydra-SGG: Hybrid Relation Assignment for One-stage Scene Graph Generation](https://mlsanity.com/paper/2409.10262/) (2024)
- [Rethinking the Evaluation of Unbiased Scene Graph Generation](https://mlsanity.com/paper/2208.01909/) (2022)
- [Environment-Invariant Curriculum Relation Learning for Fine-Grained Scene Graph Generation](https://mlsanity.com/paper/2308.03282/) (2023)
- [Rethinking the Reference-based Distinctive Image Captioning](https://mlsanity.com/paper/2207.11118/) (2022)
- [Probabilistic Debiasing of Scene Graphs](https://mlsanity.com/paper/2211.06444/) (2022)
- [Leveraging Predicate and Triplet Learning for Scene Graph Generation](https://mlsanity.com/paper/2406.02038/) (2024)
- [Enhancing Scene Graph Generation with Hierarchical Relationships and Commonsense Knowledge](https://mlsanity.com/paper/2311.12889/) (2023)

## Abstract (arXiv)

Unbiased SGG has achieved significant progress over recent years. However, almost all existing SGG models have overlooked the ground-truth annotation qualities of prevailing SGG datasets, i.e., they always assume: 1) all the manually annotated positive samples are equally correct; 2) all the un-annotated negative samples are absolutely background. In this paper, we argue that both assumptions are inapplicable to SGG: there are numerous "noisy" groundtruth predicate labels that break these two assumptions, and these noisy samples actually harm the training of unbiased SGG models. To this end, we propose a novel model-agnostic NoIsy label CorrEction strategy for SGG: NICE. NICE can not only detect noisy samples but also reassign more high-quality predicate labels to them. After the NICE training, we can obtain a cleaner version of SGG dataset for model training. Specifically, NICE consists of three components: negative Noisy Sample Detection (Neg-NSD), positive NSD (Pos-NSD), and Noisy Sample Correction (NSC). Firstly, in Neg-NSD, we formulate this task as an out-of-distribution detection problem, and assign pseudo labels to all detected noisy negative samples. Then, in Pos-NSD, we use a clustering-based algorithm to divide all positive samples into multiple sets, and treat the samples in the noisiest set as noisy positive samples. Lastly, in NSC, we use a simple but effective weighted KNN to reassign new predicate labels to noisy positive samples. Extensive results on different backbones and tasks have attested to the effectiveness and generalization abilities of each component of NICE.