# RecurGS: Interactive Scene Modeling via Discrete-State Recurrent Gaussian Fusion

arXiv: https://arxiv.org/abs/2512.18386 · HTML: https://mlsanity.com/paper/2512.18386/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 40 references that are in the map.
- First posted on arXiv 20 Dec 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023)
- [RIO: 3D Object Instance Re-Localization in Changing Indoor Environments](https://mlsanity.com/paper/1908.06109/) (2019)
- [GaussianEditor: Editing 3D Gaussians Delicately with Text Instructions](https://mlsanity.com/paper/2311.16037/) (2023)
- [3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans](https://mlsanity.com/paper/2002.06289/) (2020)
- [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024)
- [Physics3D: Learning Physical Properties of 3D Gaussians via Video Diffusion](https://mlsanity.com/paper/2406.04338/) (2024)
- [Scene Graph Generation: A Comprehensive Survey](https://mlsanity.com/paper/2201.00443/) (2022)
- [Language-Grounded Dynamic Scene Graphs for Interactive Object Search with Mobile Manipulation](https://mlsanity.com/paper/2403.08605/) (2024)
- [Fast and Robust Iterative Closest Point](https://mlsanity.com/paper/2007.07627/) (2020)
- [Advances in 3D Generation: A Survey](https://mlsanity.com/paper/2401.17807/) (2024)
- [RoboGSim: A Real2Sim2Real Robotic Gaussian Splatting Simulator](https://mlsanity.com/paper/2411.11839/) (2024)
- [Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting](https://mlsanity.com/paper/2411.09823/) (2024)
- [Decompositional Neural Scene Reconstruction with Generative Diffusion Prior](https://mlsanity.com/paper/2503.14830/) (2025)
- [CLNeRF: Continual Learning Meets NeRF](https://mlsanity.com/paper/2308.14816/) (2023)
- [EmbodiedRAG: Dynamic 3D Scene Graph Retrieval for Efficient and Scalable Robot Task Planning](https://mlsanity.com/paper/2410.23968/) (2024)
- [3D Scene Generation: A Survey](https://mlsanity.com/paper/2505.05474/) (2025)
- [PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation](https://mlsanity.com/paper/2504.16693/) (2025)
- [DecoupledGaussian: Object-Scene Decoupling for Physics-Based Interaction](https://mlsanity.com/paper/2503.05484/) (2025)
- [Reconstructing Interactive 3D Scenes by Panoptic Mapping and CAD Model Alignments](https://mlsanity.com/paper/2103.16095/) (2021)
- [Living Scenes: Multi-object Relocalization and Reconstruction in Changing 3D Environments](https://mlsanity.com/paper/2312.09138/) (2023)
- [HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video](https://mlsanity.com/paper/2510.05560/) (2025)
- [Rescan: Inductive Instance Segmentation for Indoor RGBD Scans](https://mlsanity.com/paper/1909.11268/) (2019)
- [MEIL-NeRF: Memory-Efficient Incremental Learning of Neural Radiance Fields](https://mlsanity.com/paper/2212.08328/) (2022)
- [Embodied Intelligence for 3D Understanding: A Survey on 3D Scene Question Answering](https://mlsanity.com/paper/2502.00342/) (2025)
- [GaussianUpdate: Continual 3D Gaussian Splatting Update for Changing Environments](https://mlsanity.com/paper/2508.08867/) (2025)

## Built on by (most cited first, 1 of 1)

- [LightFuse: Relightable Interactive Gaussian Scene Reconstruction via Multi-Scan Fusion and 2D Gaussian Ray Tracing](https://mlsanity.com/paper/2608.29269/) (2026)

## Abstract (arXiv)

Recent advances in 3D scene representations have enabled high-fidelity novel view synthesis, yet adapting to discrete scene changes and constructing interactive 3D environments remain open challenges in vision and robotics. Existing approaches focus solely on updating a single scene without supporting novel-state synthesis. Others rely on diffusion-based object-background decoupling that works on one state at a time and cannot fuse information across multiple observations. To address these limitations, we introduce RecurGS, a recurrent fusion framework that incrementally integrates discrete Gaussian scene states into a single evolving representation capable of interaction. RecurGS detects object-level changes across consecutive states, aligns their geometric motion using semantic correspondence and Lie-algebra based SE(3) refinement, and performs recurrent updates that preserve historical structures through replay supervision. A voxelized, visibility-aware fusion module selectively incorporates newly observed regions while keeping stable areas fixed, mitigating catastrophic forgetting and enabling efficient long-horizon updates. RecurGS supports object-level manipulation, synthesizes novel scene states without requiring additional scans, and maintains photorealistic fidelity across evolving environments. Extensive experiments across synthetic and real-world datasets demonstrate that our framework delivers high-quality reconstructions with substantially improved update efficiency, providing a scalable step toward continuously interactive Gaussian worlds.