# Gaga: Group Any Gaussians via 3D-aware Memory Bank

arXiv: https://arxiv.org/abs/2404.07977 · HTML: https://mlsanity.com/paper/2404.07977/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 46 references that are in the map.
- First posted on arXiv 11 Apr 2024.

## Builds on (most cited first, 30 of 35)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [TensoRF: Tensorial Radiance Fields](https://mlsanity.com/paper/2203.09517/) (2022)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Mip-Splatting: Alias-free 3D Gaussian Splatting](https://mlsanity.com/paper/2311.16493/) (2023)
- [Nerfstudio: A Modular Framework for Neural Radiance Field Development](https://mlsanity.com/paper/2302.04264/) (2023)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation](https://mlsanity.com/paper/2205.04334/) (2022)
- [Track Anything: Segment Anything Meets Videos](https://mlsanity.com/paper/2304.11968/) (2023)
- [A Survey on 3D Gaussian Splatting](https://mlsanity.com/paper/2401.03890/) (2024)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [Segment and Track Anything](https://mlsanity.com/paper/2305.06558/) (2023)
- [Panoptic Lifting for 3D Scene Understanding with Neural Fields](https://mlsanity.com/paper/2212.09802/) (2022)
- [Segment Any 3D Gaussians](https://mlsanity.com/paper/2312.00860/) (2023)
- [Panoptic NeRF: 3D-to-2D Label Transfer for Panoptic Urban Scene Segmentation](https://mlsanity.com/paper/2203.15224/) (2022)
- [Weakly Supervised 3D Open-vocabulary Segmentation](https://mlsanity.com/paper/2305.14093/) (2023)
- [Recent Advances in 3D Gaussian Splatting](https://mlsanity.com/paper/2403.11134/) (2024)
- [GARField: Group Anything with Radiance Fields](https://mlsanity.com/paper/2401.09419/) (2024)
- [High-Quality Entity Segmentation](https://mlsanity.com/paper/2211.05776/) (2022)

## Built on by (most cited first, 30 of 41)

- [InstanceGaussian: Appearance-Semantic Joint Gaussian Representation for 3D Instance-Level Perception](https://mlsanity.com/paper/2411.19235/) (2024)
- [Rethinking End-to-End 2D to 3D Scene Segmentation in Gaussian Splatting](https://mlsanity.com/paper/2503.14029/) (2025)
- [ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splatting](https://mlsanity.com/paper/2507.15454/) (2025)
- [Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentation](https://mlsanity.com/paper/2502.00173/) (2025)
- [A Survey on 3D Gaussian Splatting Applications: Segmentation, Editing, and Generation](https://mlsanity.com/paper/2508.09977/) (2025)
- [SLGaussian: Fast Language Gaussian Splatting in Sparse Views](https://mlsanity.com/paper/2412.08331/) (2024)
- [Tackling View-Dependent Semantics in 3D Language Gaussian Splatting](https://mlsanity.com/paper/2505.24746/) (2025)
- [Wheat3DGS: In-field 3D Reconstruction, Instance Segmentation and Phenotyping of Wheat Heads with Gaussian Splatting](https://mlsanity.com/paper/2504.06978/) (2025)
- [Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracing](https://mlsanity.com/paper/2508.03227/) (2025)
- [TRASE: Tracking-free 4D Segmentation and Editing](https://mlsanity.com/paper/2411.19290/) (2024)
- [Inpaint360GS: Efficient Object-Aware 3D Inpainting via Gaussian Splatting for 360deg Scenes](https://mlsanity.com/paper/2511.06457/) (2025)
- [Pointmap Association and Piecewise-Plane Constraint for Consistent and Compact 3D Gaussian Segmentation Field](https://mlsanity.com/paper/2502.16303/) (2025)
- [REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting](https://mlsanity.com/paper/2510.16410/) (2025)
- [Bayesian Fields: Task-driven Open-Set Semantic Gaussian Splatting](https://mlsanity.com/paper/2503.05949/) (2025)
- [DCHM: Depth-Consistent Human Modeling for Multiview Detection](https://mlsanity.com/paper/2507.14505/) (2025)
- [Consistent Instance Field for Dynamic Scene Understanding](https://mlsanity.com/paper/2512.14126/) (2025)
- [OpenVoxel: Training-Free Grouping and Captioning Voxels for Open-Vocabulary 3D Scene Understanding](https://mlsanity.com/paper/2601.09575/) (2026)
- [Gradient-Weighted Feature Back-Projection: A Fast Alternative to Feature Distillation in 3D Gaussian Splatting](https://mlsanity.com/paper/2411.15193/) (2024)
- [MGSR: 2D/3D Mutual-boosted Gaussian Splatting for High-fidelity Surface Reconstruction under Various Light Conditions](https://mlsanity.com/paper/2503.05182/) (2025)
- [OVSeg3R: Learn Open-vocabulary Instance Segmentation from 2D via 3D Reconstruction](https://mlsanity.com/paper/2509.23541/) (2025)
- [Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation](https://mlsanity.com/paper/2512.22745/) (2025)
- [Space-Time Forecasting of Dynamic Scenes with Motion-aware Gaussian Grouping](https://mlsanity.com/paper/2602.21668/) (2026)
- [Enforcing View-Consistency in Class-Agnostic 3D Segmentation Fields](https://mlsanity.com/paper/2408.09928/) (2024)
- [IGFuse: Interactive 3D Gaussian Scene Reconstruction via Multi-Scans Fusion](https://mlsanity.com/paper/2508.13153/) (2025)
- [B3-Seg: Camera-Free, Training-Free 3DGS Segmentation via Analytic EIG and Beta-Bernoulli Bayesian Updates](https://mlsanity.com/paper/2602.17134/) (2026)
- [WorldAct: Activating Monolithic 3D Worlds into Interactive-Ready Object-Centric Scenes](https://mlsanity.com/paper/2605.15843/) (2026)
- [DSG-World: Learning a 3D Gaussian World Model from Dual State Videos](https://mlsanity.com/paper/2506.05217/) (2025)
- [CoRe-GS: Coarse-to-Refined Gaussian Splatting with Semantic Object Focus](https://mlsanity.com/paper/2509.04859/) (2025)
- [TIBR4D: Tracing-Guided Iterative Boundary Refinement for Efficient 4D Gaussian Segmentation](https://mlsanity.com/paper/2602.08540/) (2026)
- [GASE: Gaussian Splatting-Based Automated System for Reconstructing Embodied-Simulation Environments](https://mlsanity.com/paper/2606.17520/) (2026)

## Abstract (arXiv)

We introduce Gaga, a framework that reconstructs and segments open-world 3D scenes by leveraging inconsistent 2D masks predicted by zero-shot class-agnostic segmentation models. Contrasted to prior 3D scene segmentation approaches that rely on video object tracking or contrastive learning methods, Gaga utilizes spatial information and effectively associates object masks across diverse camera poses through a novel 3D-aware memory bank. By eliminating the assumption of continuous view changes in training images, Gaga demonstrates robustness to variations in camera poses, particularly beneficial for sparsely sampled images, ensuring precise mask label consistency. Furthermore, Gaga accommodates 2D segmentation masks from diverse sources and demonstrates robust performance with different open-world zero-shot class-agnostic segmentation models, significantly enhancing its versatility. Extensive qualitative and quantitative evaluations demonstrate that Gaga performs favorably against state-of-the-art methods, emphasizing its potential for real-world applications such as 3D scene understanding and manipulation.