Scenes as Objects, Not Primitives: Instance-Structured 3D Tokenization from Unposed Views
Not on the map. Why?
Abstract (arXiv)
A 3D scene is understood through its objects, not the primitives that compose them. Yet feed-forward reconstruction methods output dense, unstructured sets of points or Gaussians, leaving object-level structure to be recovered after the fact. We propose a feed-forward framework that decomposes a scene into instance-structured 3D token groups directly from unposed multi-view images -- compact object-centric units from which reconstruction, segmentation, and manipulation all follow. Each token group pairs an instance token capturing entity-level identity with anchor tokens that encode local geometry and appearance, which are decoded into a set of 3D Gaussians. This two-level factorization decouples object identity from local appearance, making object instances a native interface of the representation rather than a derived product. The token groups are learned through differentiable rendering with joint reconstruction and segmentation supervision, requiring no 3D annotations. Our feed-forward model surpasses per-scene optimization baselines in class-agnostic instance segmentation while remaining competitive in novel view synthesis. Beyond these metrics, the same token groups directly unlock instance-level scene editing -- removing, translating, or inserting objects by operating on their groups -- as well as efficient open-vocabulary 3D instance retrieval, where retrieval complexity scales with the number of instances rather than primitives.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 36 references that are in the map. First posted on arXiv 28 Jun 2026.
Builds on
34 of 36 references.
References · 34
- 2020End-to-End Object Detection with Transformerscited by 7,384
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,088
- 2024SAM 2: Segment Anything in Images and Videoscited by 2,784
- 2021Masked-attention Mask Transformer for Universal Image Segmentationcited by 2,338
- 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
- 2021Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fieldscited by 2,013
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
- 2025VGGT: Visual Geometry Grounded Transformercited by 1,475
- 2024Grounding Image Matching in 3D with MASt3Rcited by 960
- 2020Object-Centric Learning with Slot Attentioncited by 865
- 2022Language-driven Semantic Segmentationcited by 711
22 more
- 2023pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstructioncited by 668
- 2024MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Imagescited by 553
- 2023Gaussian Grouping: Segment and Edit Anything in 3D Scenescited by 387
- 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
- 2024gsplat: An Open-Source Library for Gaussian Splattingcited by 279
- 2024No Pose, No Problem: Surprisingly Simple 3D Gaussian Splats from Sparse Unposed Imagescited by 241
- 2025AnySplat: Feed-forward 3D Gaussian Splatting from Unconstrained Viewscited by 234
- 2024Splatt3R: Zero-shot Gaussian Splatting from Uncalibrated Image Pairscited by 171
- 2023SAM3D: Segment Anything in 3D Scenescited by 155
- 2024Large Spatial Model: End-to-end Unposed Images to Semantic 3Dcited by 73
- 2024LUDVIG: Learning-Free Uplifting of 2D Visual Features to Gaussian Splatting Scenescited by 36
- 2025IGGT: Instance-Grounded Geometry Transformer for Semantic 3D Reconstructioncited by 26
- 2025Uni3R: Unified 3D Reconstruction and Semantic Understanding via Generalizable Gaussian Splatting from Unposed Multi-View Imagescited by 25
- 2025ObjectGS: Object-aware Scene Reconstruction and Scene Understanding via Gaussian Splattingcited by 24
- 2025Lifting by Gaussians: A Simple, Fast and Flexible Method for 3D Instance Segmentationcited by 23
- 2025PanSt3R: Multi-view Consistent Panoptic Segmentationcited by 19
- 2025C3G: Learning Compact 3D Representations with 2K Gaussianscited by 18
- 2024Any3DIS: Class-Agnostic 3D Instance Segmentation by 2D Mask Trackingcited by 14
- 2025Trace3D: Consistent Segmentation Lifting via Gaussian Instance Tracingcited by 13
- 2022kMaX-DeepLab: k-means Mask Transformercited by 12
- 2024SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fieldscited by 12
- 2026Scene-Agnostic Object-Centric Representation Learning for 3D Gaussian Splattingcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.