# 3D-Aware Scene Manipulation via Inverse Graphics

arXiv: https://arxiv.org/abs/1808.09351 · HTML: https://mlsanity.com/paper/1808.09351/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Aug 2018.

## Built on by (most cited first, 24 of 24)

- [SPIn-NeRF: Multiview Segmentation and Perceptual Inpainting with Neural Radiance Fields](https://mlsanity.com/paper/2211.12254/) (2022)
- [Unsupervised Discovery of Object Radiance Fields](https://mlsanity.com/paper/2107.07905/) (2021)
- [LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding](https://mlsanity.com/paper/2312.14074/) (2023)
- [Reference-guided Controllable Inpainting of Neural Radiance Fields](https://mlsanity.com/paper/2304.09677/) (2023)
- [Learning Object-Centric Representations of Multi-Object Scenes from Multiple Views](https://mlsanity.com/paper/2111.07117/) (2021)
- [BlobGAN: Spatially Disentangled Scene Representations](https://mlsanity.com/paper/2205.02837/) (2022)
- [Neural Material Adaptor for Visual Grounding of Intrinsic Dynamics](https://mlsanity.com/paper/2410.08257/) (2024)
- [Compositional Scene Representation Learning via Reconstruction: A Survey](https://mlsanity.com/paper/2202.07135/) (2022)
- [3D-aware Conditional Image Synthesis](https://mlsanity.com/paper/2302.08509/) (2023)
- [SPARF: Large-Scale Learning of 3D Sparse Radiance Fields from Few Input Images](https://mlsanity.com/paper/2212.09100/) (2022)
- [Customizing Text-to-Image Diffusion with Object Viewpoint Control](https://mlsanity.com/paper/2404.12333/) (2024)
- [Unsupervised Discovery of Object-Centric Neural Fields](https://mlsanity.com/paper/2402.07376/) (2024)
- [Deep Feedback Inverse Problem Solver](https://mlsanity.com/paper/2101.07719/) (2021)
- [iSegMan: Interactive Segment-and-Manipulate 3D Gaussians](https://mlsanity.com/paper/2505.11934/) (2025)
- [Learning Scene Dynamics from Point Cloud Sequences](https://mlsanity.com/paper/2111.08755/) (2021)
- [Inverse Rendering of Translucent Objects using Physical and Neural Renderers](https://mlsanity.com/paper/2305.08336/) (2023)
- [Thinking with Spatial Code for Physical-World Video Reasoning](https://mlsanity.com/paper/2603.05591/) (2026)
- [Inferring Fluid Dynamics via Inverse Rendering](https://mlsanity.com/paper/2304.04446/) (2023)
- [GANs Conditioning Methods: A Survey](https://mlsanity.com/paper/2408.15640/) (2024)
- [Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Models](https://mlsanity.com/paper/2608.25518/) (2026)
- [Robust 2D/3D Vehicle Parsing in CVIS](https://mlsanity.com/paper/2103.06432/) (2021)
- [Object Pursuit: Building a Space of Objects via Discriminative Weight Generation](https://mlsanity.com/paper/2112.07954/) (2021)
- [Visual Question Answering From Another Perspective: CLEVR Mental Rotation Tests](https://mlsanity.com/paper/2212.01639/) (2022)
- [A Safety Assurable Human-Inspired Perception Architecture](https://mlsanity.com/paper/2205.07862/) (2022)

## Abstract (arXiv)

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.