# 4DSegStreamer: Streaming 4D Panoptic Segmentation via Dual Threads

arXiv: https://arxiv.org/abs/2510.17664 · HTML: https://mlsanity.com/paper/2510.17664/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 38 of its 49 references that are in the map.
- First posted on arXiv 20 Oct 2025.

## Builds on (most cited first, 30 of 38)

- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Point Transformer V3: Simpler, Faster, Stronger](https://mlsanity.com/paper/2312.10035/) (2023)
- [DriveVLM: The Convergence of Autonomous Driving and Large Vision-Language Models](https://mlsanity.com/paper/2402.12289/) (2024)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Delving Deeper into Convolutional Networks for Learning Video Representations](https://mlsanity.com/paper/1511.06432/) (2015)
- [SuMa++: Efficient LiDAR-based Semantic SLAM](https://mlsanity.com/paper/2105.11320/) (2021)
- [4D Panoptic LiDAR Segmentation](https://mlsanity.com/paper/2102.12472/) (2021)
- [Fast Neural Scene Flow](https://mlsanity.com/paper/2304.09121/) (2023)
- [SODFormer: Streaming Object Detection with Transformer Using Events and Frames](https://mlsanity.com/paper/2308.04047/) (2023)
- [Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding](https://mlsanity.com/paper/2208.00281/) (2022)
- [4D-StOP: Panoptic Segmentation of 4D LiDAR using Spatio-temporal Object Proposal Generation and Aggregation](https://mlsanity.com/paper/2209.14858/) (2022)
- [Real-time Object Detection for Streaming Perception](https://mlsanity.com/paper/2203.12338/) (2022)
- [Are We Ready for Vision-Centric Driving Streaming Perception? The ASAP Benchmark](https://mlsanity.com/paper/2212.08914/) (2022)
- [Mask4Former: Mask Transformer for 4D Panoptic Segmentation](https://mlsanity.com/paper/2309.16133/) (2023)
- [4D Panoptic Segmentation as Invariant and Equivariant Field Prediction](https://mlsanity.com/paper/2303.15651/) (2023)
- [MarS3D: A Plug-and-Play Motion-Aware Model for Semantic Segmentation on Multi-Scan 3D Point Clouds](https://mlsanity.com/paper/2307.09316/) (2023)
- [FASIONAD : FAst and Slow FusION Thinking Systems for Human-Like Autonomous Driving with Adaptive Feedback](https://mlsanity.com/paper/2411.18013/) (2024)
- [Streaming Object Detection for 3-D Point Clouds](https://mlsanity.com/paper/2005.01864/) (2020)
- [StrObe: Streaming Object Detection from LiDAR Packets](https://mlsanity.com/paper/2011.06425/) (2020)
- [Position-Guided Point Cloud Panoptic Segmentation Transformer](https://mlsanity.com/paper/2303.13509/) (2023)
- [ZeroFlow: Scalable Scene Flow via Distillation](https://mlsanity.com/paper/2305.10424/) (2023)
- [SVQNet: Sparse Voxel-Adjacent Query Network for 4D Spatio-Temporal LiDAR Semantic Segmentation](https://mlsanity.com/paper/2308.13323/) (2023)
- [MemorySeg: Online LiDAR Semantic Segmentation with a Latent Memory](https://mlsanity.com/paper/2311.01556/) (2023)
- [DeepTemporalSeg: Temporally Consistent Semantic Segmentation of 3D LiDAR Scans](https://mlsanity.com/paper/1906.06962/) (2019)
- [DAMO-StreamNet: Optimizing Streaming Perception in Autonomous Driving](https://mlsanity.com/paper/2303.17144/) (2023)
- [Memory-based Adapters for Online 3D Scene Perception](https://mlsanity.com/paper/2403.06974/) (2024)
- [TASeg: Temporal Aggregation Network for LiDAR Semantic Segmentation](https://mlsanity.com/paper/2407.09751/) (2024)
- [LongShortNet: Exploring Temporal and Semantic Features Fusion in Streaming Perception](https://mlsanity.com/paper/2210.15518/) (2022)
- [Context-Aware Streaming Perception in Dynamic Environments](https://mlsanity.com/paper/2208.07479/) (2022)
- [Chanakya: Learning Runtime Decisions for Adaptive Real-Time Perception](https://mlsanity.com/paper/2106.05665/) (2021)

## Abstract (arXiv)

4D panoptic segmentation in a streaming setting is critical for highly dynamic environments, such as evacuating dense crowds and autonomous driving in complex scenarios, where real-time, fine-grained perception within a constrained time budget is essential. In this paper, we introduce 4DSegStreamer, a novel framework that employs a Dual-Thread System to efficiently process streaming frames. The framework is general and can be seamlessly integrated into existing 3D and 4D segmentation methods to enable real-time capability. It also demonstrates superior robustness compared to existing streaming perception approaches, particularly under high FPS conditions. The system consists of a predictive thread and an inference thread. The predictive thread leverages historical motion and geometric information to extract features and forecast future dynamics. The inference thread ensures timely prediction for incoming frames by aligning with the latest memory and compensating for ego-motion and dynamic object movements. We evaluate 4DSegStreamer on the indoor HOI4D dataset and the outdoor SemanticKITTI and nuScenes datasets. Comprehensive experiments demonstrate the effectiveness of our approach, particularly in accurately predicting dynamic objects in complex scenes.