# Hierarchical Temporal Context Learning for Camera-based Semantic Scene Completion

arXiv: https://arxiv.org/abs/2407.02077 · HTML: https://mlsanity.com/paper/2407.02077/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 54 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Builds on (most cited first, 30 of 38)

- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Deformable Convolutional Networks](https://mlsanity.com/paper/1703.06211/) (2017)
- [Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D](https://mlsanity.com/paper/2008.05711/) (2020)
- [The Replica Dataset: A Digital Replica of Indoor Spaces](https://mlsanity.com/paper/1906.05797/) (2019)
- [BEVDet: High-performance Multi-camera 3D Object Detection in Bird-Eye-View](https://mlsanity.com/paper/2112.11790/) (2021)
- [AdaBins: Depth Estimation using Adaptive Bins](https://mlsanity.com/paper/2011.14141/) (2020)
- [BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection](https://mlsanity.com/paper/2206.10092/) (2022)
- [PETR: Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2203.05625/) (2022)
- [Semantic Scene Completion from a Single Depth Image](https://mlsanity.com/paper/1611.08974/) (2016)
- [3D Packing for Self-Supervised Monocular Depth Estimation](https://mlsanity.com/paper/1905.02693/) (2019)
- [Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2302.07817/) (2023)
- [MonoScene: Monocular 3D Semantic Scene Completion](https://mlsanity.com/paper/2112.00726/) (2021)
- [PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images](https://mlsanity.com/paper/2206.01256/) (2022)
- [SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving](https://mlsanity.com/paper/2303.09551/) (2023)
- [VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion](https://mlsanity.com/paper/2302.12251/) (2023)
- [OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction](https://mlsanity.com/paper/2304.05316/) (2023)
- [Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion](https://mlsanity.com/paper/2012.03762/) (2020)
- [OpenOccupancy: A Large Scale Benchmark for Surrounding Semantic Occupancy Perception](https://mlsanity.com/paper/2303.03991/) (2023)
- [Consistent Video Depth Estimation](https://mlsanity.com/paper/2004.15021/) (2020)
- [PatchmatchNet: Learned Multi-View Patchmatch Stereo](https://mlsanity.com/paper/2012.01411/) (2020)
- [The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth](https://mlsanity.com/paper/2104.14540/) (2021)
- [LMSCNet: Lightweight Multiscale 3D Semantic Completion](https://mlsanity.com/paper/2008.10559/) (2020)
- [Robust Consistent Video Depth Estimation](https://mlsanity.com/paper/2012.05901/) (2020)
- [Hierarchical Neural Architecture Search for Deep Stereo Matching](https://mlsanity.com/paper/2010.13501/) (2020)
- [Sparse4D: Multi-view 3D Object Detection with Sparse Spatial-Temporal Fusion](https://mlsanity.com/paper/2211.10581/) (2022)
- [Anisotropic Convolutional Networks for 3D Semantic Scene Completion](https://mlsanity.com/paper/2004.02122/) (2020)
- [3D Sketch-aware Semantic Scene Completion via Semi-supervised Structure Prior](https://mlsanity.com/paper/2003.14052/) (2020)
- [S3CNet: A Sparse Semantic Scene Completion Network for LiDAR Point Clouds](https://mlsanity.com/paper/2012.09242/) (2020)
- [Semantic Scene Completion using Local Deep Implicit Functions on LiDAR Data](https://mlsanity.com/paper/2011.09141/) (2020)

## Built on by (most cited first, 30 of 32)

- [DiST-4D: Disentangled Spatiotemporal Diffusion with Metric Depth for 4D Driving Scene Generation](https://mlsanity.com/paper/2503.15208/) (2025)
- [OmniNWM: Omniscient Driving Navigation World Models](https://mlsanity.com/paper/2510.18313/) (2025)
- [GaussRender: Learning 3D Occupancy with Gaussian Rendering](https://mlsanity.com/paper/2502.05040/) (2025)
- [ORV: 4D Occupancy-centric Robot Video Generation](https://mlsanity.com/paper/2506.03079/) (2025)
- [Challenger: Affordable Adversarial Driving Video Generation](https://mlsanity.com/paper/2505.15880/) (2025)
- [Unifying Appearance Codes and Bilateral Grids for Driving Scene Gaussian Splatting](https://mlsanity.com/paper/2506.05280/) (2025)
- [VoxDet: Rethinking 3D Semantic Occupancy Prediction as Dense Object Detection](https://mlsanity.com/paper/2506.04623/) (2025)
- [OccScene: Semantic Occupancy-based Cross-task Mutual Learning for 3D Scene Generation](https://mlsanity.com/paper/2412.11183/) (2024)
- [Hierarchical Context Alignment with Disentangled Geometric and Temporal Modeling for Semantic Occupancy Prediction](https://mlsanity.com/paper/2412.08243/) (2024)
- [MuDG: Taming Multi-modal Diffusion with Gaussian Splatting for Urban Scene Reconstruction](https://mlsanity.com/paper/2503.10604/) (2025)
- [4D Driving Scene Generation With Stereo Forcing](https://mlsanity.com/paper/2509.20251/) (2025)
- [Scaling Up Occupancy-centric Driving Scene Generation: Dataset and Method](https://mlsanity.com/paper/2510.22973/) (2025)
- [Disentangling Instance and Scene Contexts for 3D Semantic Scene Completion](https://mlsanity.com/paper/2507.08555/) (2025)
- [VisHall3D: Monocular Semantic Scene Completion from Reconstructing the Visible Regions to Hallucinating the Invisible Regions](https://mlsanity.com/paper/2507.19188/) (2025)
- [Hybrid-grained Feature Aggregation with Coarse-to-fine Language Guidance for Self-supervised Monocular Depth Estimation](https://mlsanity.com/paper/2510.09320/) (2025)
- [Spatial Retrieval Augmented Autonomous Driving](https://mlsanity.com/paper/2512.06865/) (2025)
- [Three Cars Approaching within 100m! Enhancing Distant Geometry by Tri-Axis Voxel Scanning for Camera-based Semantic Scene Completion](https://mlsanity.com/paper/2411.16129/) (2024)
- [Event-aided Semantic Scene Completion](https://mlsanity.com/paper/2502.02334/) (2025)
- [Learning Temporal 3D Semantic Scene Completion via Optical Flow Guidance](https://mlsanity.com/paper/2502.14520/) (2025)
- [One Step Closer: Creating the Future to Boost Monocular Semantic Scene Completion](https://mlsanity.com/paper/2507.13801/) (2025)
- [AdaSFormer: Adaptive Serialized Transformers for Monocular Semantic Scene Completion from Indoor Environments](https://mlsanity.com/paper/2603.25494/) (2026)
- [Test-Time 3D Occupancy Prediction](https://mlsanity.com/paper/2503.08485/) (2025)
- [Out-of-Distribution Semantic Occupancy Prediction](https://mlsanity.com/paper/2506.21185/) (2025)
- [Unleashing Semantic and Geometric Priors for 3D Scene Completion](https://mlsanity.com/paper/2508.13601/) (2025)
- [Seen2Scene: Completing Realistic 3D Scenes with Visibility-Guided Flow](https://mlsanity.com/paper/2603.28548/) (2026)
- [Bridging 3D Gaussians and Semantic Occupancy for Comprehensive Open-Vocabulary Scene Understanding from Unposed Images](https://mlsanity.com/paper/2607.01633/) (2026)
- [Towards Temporal Fusion Beyond the Field of View for Camera-based Semantic Scene Completion](https://mlsanity.com/paper/2511.12498/) (2025)
- [Towards 3D Object-Centric Feature Learning for Semantic Scene Completion](https://mlsanity.com/paper/2511.13031/) (2025)
- [VOIC: Visible-Occluded Integrated Guidance for 3D Semantic Scene Completion](https://mlsanity.com/paper/2512.18954/) (2025)
- [SPOT-Occ: Sparse Prototype-guided Transformer for Camera-based 3D Occupancy Prediction](https://mlsanity.com/paper/2602.04240/) (2026)

## Abstract (arXiv)

Camera-based 3D semantic scene completion (SSC) is pivotal for predicting complicated 3D layouts with limited 2D image observations. The existing mainstream solutions generally leverage temporal information by roughly stacking history frames to supplement the current frame, such straightforward temporal modeling inevitably diminishes valid clues and increases learning difficulty. To address this problem, we present HTCL, a novel Hierarchical Temporal Context Learning paradigm for improving camera-based semantic scene completion. The primary innovation of this work involves decomposing temporal context learning into two hierarchical steps: (a) cross-frame affinity measurement and (b) affinity-based dynamic refinement. Firstly, to separate critical relevant context from redundant information, we introduce the pattern affinity with scale-aware isolation and multiple independent learners for fine-grained contextual correspondence modeling. Subsequently, to dynamically compensate for incomplete observations, we adaptively refine the feature sampling locations based on initially identified locations with high affinity and their neighboring relevant regions. Our method ranks 1st on the SemanticKITTI benchmark and even surpasses LiDAR-based methods in terms of mIoU on the OpenOccupancy benchmark. Our code is available on https://github.com/Arlo0o/HTCL.