# Joint Semantic and Rendering Enhancements in 3D Gaussian Modeling with Anisotropic Local Encoding

arXiv: https://arxiv.org/abs/2601.02339 · HTML: https://mlsanity.com/paper/2601.02339/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 5 Jan 2026.

## Builds on (most cited first, 30 of 37)

- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [Language-driven Semantic Segmentation](https://mlsanity.com/paper/2201.03546/) (2022)
- [Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering](https://mlsanity.com/paper/2312.00109/) (2023)
- [LERF: Language Embedded Radiance Fields](https://mlsanity.com/paper/2303.09553/) (2023)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [In-Place Scene Labelling and Understanding with Implicit Scene Representation](https://mlsanity.com/paper/2103.15875/) (2021)
- [LangSplat: 3D Language Gaussian Splatting](https://mlsanity.com/paper/2312.16084/) (2023)
- [Gaussian Grouping: Segment and Edit Anything in 3D Scenes](https://mlsanity.com/paper/2312.00732/) (2023)
- [Compact 3D Gaussian Representation for Radiance Field](https://mlsanity.com/paper/2311.13681/) (2023)
- [Decomposing NeRF for Editing via Feature Field Distillation](https://mlsanity.com/paper/2205.15585/) (2022)
- [Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields](https://mlsanity.com/paper/2312.03203/) (2023)
- [GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting](https://mlsanity.com/paper/2311.14521/) (2023)
- [Language Embedded 3D Gaussians for Open-Vocabulary Scene Understanding](https://mlsanity.com/paper/2311.18482/) (2023)
- [GaussianPro: 3D Gaussian Splatting with Progressive Propagation](https://mlsanity.com/paper/2402.14650/) (2024)
- [Tracking Anything with Decoupled Video Segmentation](https://mlsanity.com/paper/2309.03903/) (2023)
- [EAGLES: Efficient Accelerated 3D Gaussians with Lightweight EncodingS](https://mlsanity.com/paper/2312.04564/) (2023)
- [Panoptic Lifting for 3D Scene Understanding with Neural Fields](https://mlsanity.com/paper/2212.09802/) (2022)
- [GaussianFormer: Scene as Gaussians for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2405.17429/) (2024)
- [HUGS: Holistic Urban 3D Scene Understanding via Gaussian Splatting](https://mlsanity.com/paper/2403.12722/) (2024)
- [Multi-Scale 3D Gaussian Splatting for Anti-Aliased Rendering](https://mlsanity.com/paper/2311.17089/) (2023)
- [Revising Densification in Gaussian Splatting](https://mlsanity.com/paper/2404.06109/) (2024)
- [FreGS: 3D Gaussian Splatting with Progressive Frequency Regularization](https://mlsanity.com/paper/2403.06908/) (2024)
- [FMGS: Foundation Model Embedded 3D Gaussian Splatting for Holistic 3D Scene Understanding](https://mlsanity.com/paper/2401.01970/) (2024)
- [Semantic Gaussians: Open-Vocabulary Scene Understanding with 3D Gaussian Splatting](https://mlsanity.com/paper/2403.15624/) (2024)
- [End-to-End Rate-Distortion Optimized 3D Gaussian Representation](https://mlsanity.com/paper/2406.01597/) (2024)
- [DM-NeRF: 3D Scene Geometry Decomposition and Manipulation from 2D Images](https://mlsanity.com/paper/2208.07227/) (2022)

## Built on by (most cited first, 2 of 2)

- [DLGStream: Dynamic Language-embedded Guassian Splatting for Open-vocabulary Enabled Free-viewpoint Video Streaming](https://mlsanity.com/paper/2606.28840/) (2026)
- [Learning Adaptive Semantic Gaussian Allocation for 3D Occupancy](https://mlsanity.com/paper/2607.21896/) (2026)

## Abstract (arXiv)

Recent works propose extending 3DGS with semantic feature vectors for simultaneous semantic segmentation and image rendering. However, these methods often treat the semantic and rendering branches separately, relying solely on 2D supervision while ignoring the 3D Gaussian geometry. Moreover, current adaptive strategies adapt the Gaussian set depending solely on rendering gradients, which can be insufficient in subtle or textureless regions. In this work, we propose a joint enhancement framework for 3D semantic Gaussian modeling that synergizes both semantic and rendering branches. Firstly, unlike conventional point cloud shape encoding, we introduce an anisotropic 3D Gaussian Chebyshev descriptor using the Laplace-Beltrami operator to capture fine-grained 3D shape details, thereby distinguishing objects with similar appearances and reducing reliance on potentially noisy 2D guidance. In addition, without relying solely on rendering gradient, we adaptively adjust Gaussian allocation and spherical harmonics with local semantic and shape signals, enhancing rendering efficiency through selective resource allocation. Finally, we employ a cross-scene knowledge transfer module to continuously update learned shape patterns, enabling faster convergence and robust representations without relearning shape information from scratch for each new scene. Experiments on multiple datasets demonstrate improvements in segmentation accuracy and rendering quality while maintaining high rendering frame rates.