# Adaptive Visual Scene Understanding: Incremental Scene Graph Generation arXiv: https://arxiv.org/abs/2310.01636 ยท HTML: https://mlsanity.com/paper/2310.01636/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 68 of its 91 references that are in the map. - First posted on arXiv 2 Oct 2023. ## Builds on (most cited first, 30 of 68) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021) - [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014) - [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015) - [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020) - [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016) - [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020) - [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023) - [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016) - [Gradient Episodic Memory for Continual Learning](https://mlsanity.com/paper/1706.08840/) (2017) - [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019) - [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019) - [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017) - [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021) - [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017) - [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018) - [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021) - [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017) - [GLIGEN: Open-Set Grounded Text-to-Image Generation](https://mlsanity.com/paper/2301.07093/) (2023) - [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018) - [Lifelong Learning with Dynamically Expandable Networks](https://mlsanity.com/paper/1708.01547/) (2017) - [RegionCLIP: Region-based Language-Image Pretraining](https://mlsanity.com/paper/2112.09106/) (2021) - [End-to-End Incremental Learning](https://mlsanity.com/paper/1807.09536/) (2018) - [The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale](https://mlsanity.com/paper/1811.00982/) (2018) - [Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interference](https://mlsanity.com/paper/1810.11910/) (2018) - [Scene Graph Generation by Iterative Message Passing](https://mlsanity.com/paper/1701.02426/) (2017) - [Visual Relationship Detection with Language Priors](https://mlsanity.com/paper/1608.00187/) (2016) ## Built on by (most cited first, 8 of 8) - [Peering into the Unknown: Active View Selection with Neural Uncertainty Maps for 3D Reconstruction](https://mlsanity.com/paper/2506.14856/) (2025) - [Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation](https://mlsanity.com/paper/2410.03858/) (2024) - [Learning to See Through a Baby's Eyes: Early Visual Diets Enable Robust Visual Intelligence in Humans and Machines](https://mlsanity.com/paper/2511.14440/) (2025) - [Learning to Perceive "Where": Spatial Pretext Tasks for Robust Self-Supervised Learning](https://mlsanity.com/paper/2605.09963/) (2026) - [CSA-Graphs: A Privacy-Preserving Structural Dataset for Child Sexual Abuse Research](https://mlsanity.com/paper/2604.07132/) (2026) - [Personalized Emotional Intelligence in Generative AI through Symbolic Affective Reasoning](https://mlsanity.com/paper/2607.10678/) (2026) - [HyperDCM: Dynamic Cluster Memory Replay in Hyperbolic Space for Continual Robotic Navigation Across Scenes](https://mlsanity.com/paper/2607.16267/) (2026) - [Remember, Revisit, Revise: Scaffold Memory for Long-Horizon 3D Geometry](https://mlsanity.com/paper/2610.19674/) (2026) ## Abstract (arXiv) Scene graph generation (SGG) analyzes images to extract meaningful information about objects and their relationships. In the dynamic visual world, it is crucial for AI systems to continuously detect new objects and establish their relationships with existing ones. Recently, numerous studies have focused on continual learning within the domains of object detection and image recognition. However, a limited amount of research focuses on a more challenging continual learning problem in SGG. This increased difficulty arises from the intricate interactions and dynamic relationships among objects, and their associated contexts. Thus, in continual learning, SGG models are often required to expand, modify, retain, and reason scene graphs within the process of adaptive visual scene understanding. To systematically explore Continual Scene Graph Generation (CSEGG), we present a comprehensive benchmark comprising three learning regimes: relationship incremental, scene incremental, and relationship generalization. Moreover, we introduce a ``Replays via Analysis by Synthesis" method named RAS. This approach leverages the scene graphs, decomposes and re-composes them to represent different scenes, and replays the synthesized scenes based on these compositional scene graphs. The replayed synthesized scenes act as a means to practice and refine proficiency in SGG in known and unknown environments. Our experimental results not only highlight the challenges of directly combining existing continual learning methods with SGG backbones but also demonstrate the effectiveness of our proposed approach, enhancing CSEGG efficiency while simultaneously preserving privacy and memory usage. All data and source code are publicly available online.