# LiteReality: Graphics-Ready 3D Scene Reconstruction from RGB-D Scans

arXiv: https://arxiv.org/abs/2507.02861 · HTML: https://mlsanity.com/paper/2507.02861/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 52 of its 69 references that are in the map.
- First posted on arXiv 3 Jul 2025.

## Builds on (most cited first, 30 of 52)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [VirtualHome: Simulating Household Activities via Programs](https://mlsanity.com/paper/1806.07011/) (2018)
- [ProcTHOR: Large-Scale Embodied AI Using Procedural Generation](https://mlsanity.com/paper/2206.06994/) (2022)
- [DISN: Deep Implicit Surface Network for High-quality Single-view 3D Reconstruction](https://mlsanity.com/paper/1905.10711/) (2019)
- [3D-FUTURE: 3D Furniture shape with TextURE](https://mlsanity.com/paper/2009.09633/) (2020)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Holodeck: Language Guided Generation of 3D Embodied AI Environments](https://mlsanity.com/paper/2312.09067/) (2023)
- [ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation](https://mlsanity.com/paper/2007.04954/) (2020)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [RoboTHOR: An Open Simulation-to-Real Embodied AI Platform](https://mlsanity.com/paper/2004.06799/) (2020)
- [Total3DUnderstanding: Joint Layout, Object Pose and Mesh Reconstruction for Indoor Scenes from a Single Image](https://mlsanity.com/paper/2002.12212/) (2020)
- [Scan2CAD: Learning CAD Model Alignment in RGB-D Scans](https://mlsanity.com/paper/1811.11187/) (2018)
- [CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image](https://mlsanity.com/paper/2502.12894/) (2025)
- [SceneScript: Reconstructing Scenes With An Autoregressive Structured Language Model](https://mlsanity.com/paper/2403.13064/) (2024)
- [Automated Creation of Digital Cousins for Robust Policy Learning](https://mlsanity.com/paper/2410.07408/) (2024)
- [IM2CAD](https://mlsanity.com/paper/1608.05137/) (2016)
- [OpenRooms: An End-to-End Open Framework for Photorealistic Indoor Scene Datasets](https://mlsanity.com/paper/2007.12868/) (2020)
- [Holistic 3D Scene Parsing and Reconstruction from a Single RGB Image](https://mlsanity.com/paper/1808.02201/) (2018)
- [ROCA: Robust CAD Model Retrieval and Alignment from a Single Image](https://mlsanity.com/paper/2112.01988/) (2021)
- [Mask2CAD: 3D Shape Prediction by Learning to Segment and Retrieve](https://mlsanity.com/paper/2007.13034/) (2020)
- [SpatialLM: Training Large Language Models for Structured Indoor Modeling](https://mlsanity.com/paper/2506.07491/) (2025)
- [Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimation](https://mlsanity.com/paper/1810.13049/) (2018)
- [DiffCAD: Weakly-Supervised Probabilistic CAD Model Retrieval and Alignment from an RGB Image](https://mlsanity.com/paper/2311.18610/) (2023)

## Built on by (most cited first, 21 of 21)

- [Articraft: An Agentic System for Scalable Articulated 3D Asset Generation](https://mlsanity.com/paper/2605.15187/) (2026)
- [PAT3D: Physics-Augmented Text-to-3D Scene Generation](https://mlsanity.com/paper/2511.21978/) (2025)
- [FunRec: Reconstructing Functional 3D Scenes from Egocentric Interaction Videos](https://mlsanity.com/paper/2604.05621/) (2026)
- [REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image](https://mlsanity.com/paper/2605.30338/) (2026)
- [UrbanVerse: Scaling Urban Simulation by Watching City-Tour Videos](https://mlsanity.com/paper/2510.15018/) (2025)
- [OmniX: From Unified Panoramic Generation and Perception to Graphics-Ready 3D Scenes](https://mlsanity.com/paper/2510.26800/) (2025)
- [PixARMesh: Autoregressive Mesh-Native Single-View Scene Reconstruction](https://mlsanity.com/paper/2603.05888/) (2026)
- [NymeriaPlus: Enriching Nymeria Dataset with Additional Annotations and Data](https://mlsanity.com/paper/2603.18496/) (2026)
- [Repurposing 3D Generative Model for Autoregressive Layout Generation](https://mlsanity.com/paper/2604.16299/) (2026)
- [FIRE3D: Feed-forward Interactive 3D Scene Reconstruction Within A Minute](https://mlsanity.com/paper/2609.08848/) (2026)
- [3D Generation for Embodied AI and Robotic Simulation: A Survey](https://mlsanity.com/paper/2604.26509/) (2026)
- [DeWorldSG: Depth-Aware 3D Semantic Scene Graph Generation via World-Model Priors](https://mlsanity.com/paper/2607.00889/) (2026)
- [SceneTeract: Probing and Improving Agent-Aware Activity Reasoning in 3D Indoor Scenes](https://mlsanity.com/paper/2603.29798/) (2026)
- [World2Minecraft: Occupancy-Driven Simulated Scenes Construction](https://mlsanity.com/paper/2604.27578/) (2026)
- [GARDEN: Gravity-Aligned Reconstruction of Disentangled ENvironments from RGB images](https://mlsanity.com/paper/2606.03921/) (2026)
- [ReScene: Structured Indoor Scene Reconstruction from Multi-View Captures](https://mlsanity.com/paper/2606.28060/) (2026)
- [Scene-SAM3D: Multi-View Scene Asset Generation Without Fine-Tuning](https://mlsanity.com/paper/2607.16805/) (2026)
- [CoDimRecon: Agentic Reconstruction of Sim-Ready 3D Scenes with Deformable Curves, Surfaces, and Volumes](https://mlsanity.com/paper/2609.36024/) (2026)
- [LEGO-Anything: Coding Agents for 3D Scene Reconstruction](https://mlsanity.com/paper/2609.36380/) (2026)
- [Scene Retargeting: Learning Object Placement with Analogical Transfer](https://mlsanity.com/paper/2609.36801/) (2026)
- [LiteReality-Agent: An Agentic System for Interactable 3D Indoor Scene Reconstruction](https://mlsanity.com/paper/2610.01863/) (2026)

## Abstract (arXiv)

We propose LiteReality, a novel pipeline that converts RGB-D scans of indoor environments into compact, realistic, and interactive 3D virtual replicas. LiteReality not only reconstructs scenes that visually resemble reality but also supports key features essential for graphics pipelines -- such as object individuality, articulation, high-quality physically based rendering materials, and physically based interaction. At its core, LiteReality first performs scene understanding and parses the results into a coherent 3D layout and objects with the help of a structured scene graph. It then reconstructs the scene by retrieving the most visually similar 3D artist-crafted models from a curated asset database. Next, the Material Painting module enhances realism by recovering high-quality, spatially varying materials. Finally, the reconstructed scene is integrated into a simulation engine with basic physical properties to enable interactive behavior. The resulting scenes are compact, editable, and fully compatible with standard graphics pipelines, making them suitable for applications in AR/VR, gaming, robotics, and digital twins. In addition, LiteReality introduces a training-free object retrieval module that achieves state-of-the-art similarity performance on the Scan2CAD benchmark, along with a robust material painting module capable of transferring appearances from images of any style to 3D assets -- even under severe misalignment, occlusion, and poor lighting. We demonstrate the effectiveness of LiteReality on both real-life scans and public datasets. Project page: https://litereality.github.io; Video: https://www.youtube.com/watch?v=ecK9m3LXg2c