# HIGS: Hierarchical Implicit Grids for Joint Geometric and Semantic Scene Understanding

arXiv: https://arxiv.org/abs/2609.38620 · HTML: https://mlsanity.com/paper/2609.38620/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 50 of its 69 references that are in the map.
- First posted on arXiv 29 Sep 2026.

## Builds on (most cited first, 30 of 50)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Instant Neural Graphics Primitives with a Multiresolution Hash Encoding](https://mlsanity.com/paper/2201.05989/) (2022)
- [ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes](https://mlsanity.com/paper/1702.04405/) (2017)
- [DeepSDF: Learning Continuous Signed Distance Functions for Shape Representation](https://mlsanity.com/paper/1901.05103/) (2019)
- [Occupancy Networks: Learning 3D Reconstruction in Function Space](https://mlsanity.com/paper/1812.03828/) (2018)
- [Plenoxels: Radiance Fields without Neural Networks](https://mlsanity.com/paper/2112.05131/) (2021)
- [Point Transformer](https://mlsanity.com/paper/2011.00931/) (2020)
- [Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression](https://mlsanity.com/paper/1902.09630/) (2019)
- [Averaging Weights Leads to Wider Optima and Better Generalization](https://mlsanity.com/paper/1803.05407/) (2018)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction](https://mlsanity.com/paper/2111.11215/) (2021)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [PlenOctrees for Real-time Rendering of Neural Radiance Fields](https://mlsanity.com/paper/2103.14024/) (2021)
- [K-Planes: Explicit Radiance Fields in Space, Time, and Appearance](https://mlsanity.com/paper/2301.10241/) (2023)
- [Open3D: A Modern Library for 3D Data Processing](https://mlsanity.com/paper/1801.09847/) (2018)
- [Implicit Geometric Regularization for Learning Shapes](https://mlsanity.com/paper/2002.10099/) (2020)
- [NICE-SLAM: Neural Implicit Scalable Encoding for SLAM](https://mlsanity.com/paper/2112.12130/) (2021)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [OpenScene: 3D Scene Understanding with Open Vocabularies](https://mlsanity.com/paper/2211.15654/) (2022)
- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Neuralangelo: High-Fidelity Neural Surface Reconstruction](https://mlsanity.com/paper/2306.03092/) (2023)
- [Scaling Open-Vocabulary Image Segmentation with Image-Level Labels](https://mlsanity.com/paper/2112.12143/) (2021)
- [ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language](https://mlsanity.com/paper/1912.08830/) (2019)
- [Neural Geometric Level of Detail: Real-time Rendering with Implicit 3D Shapes](https://mlsanity.com/paper/2101.10994/) (2021)
- [ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planning](https://mlsanity.com/paper/2309.16650/) (2023)
- [Mask3D: Mask Transformer for 3D Semantic Instance Segmentation](https://mlsanity.com/paper/2210.03105/) (2022)

## Abstract (arXiv)

Neural implicit representations have had a significant impact on scene reconstruction by enabling robots to build continuous, differentiable, and high-fidelity 3D maps. Most existing works focus on geometric reconstruction and lack semantic information for high-level spatial understanding and task planning. Also, as the scale and complexity of the environment increase, neural representations face the challenge of maintaining computational efficiency in back-end optimization. To resolve these two challenges, we introduce a hierarchical neural field that leverages multiresolution submaps to achieve an efficient and scalable implicit representation, and a unified query and decoding mechanism to support both geometric and semantic features. More specifically, the learnable map features can be converted to the output with the query and decoding process for both training and inference. For large-scale representation, we decompose a scene into overlapping submaps and do hierarchical optimization within each local submap, thus enabling scalable computation. To further improve efficiency, we design feature encoders that predict initial hierarchical grid features to substantially reduce the time needed to optimize the submap features from scratch. To correct estimation drift among submaps, we align and fuse them entirely within the implicit feature space, leading to substantial acceleration by avoiding the need to decode the final output. Building upon this efficient hierarchical representation, we embed both geometric features and vision-language latent features into the map, and demonstrate it on both Signed Distance Field (SDF) construction and open-vocabulary object grounding. Our approach significantly improves computation and memory efficiency, maintains high estimation accuracy, and endows the robot with spatial awareness on large-scale real-world benchmarks.