mlsanityData to 7 Oct 2026

4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks

arXiv 1904.08755 ↗First posted 18 Apr 2019cs.CV cs.AIcited by 1,201

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In many robotics and VR/AR applications, 3D-videos are readily-available sources of input (a continuous sequence of depth images, or LIDAR scans). However, those 3D-videos are processed frame-by-frame either through 2D convnets or 3D perception algorithms. In this work, we propose 4-dimensional convolutional neural networks for spatio-temporal perception that can directly process such 3D-videos using high-dimensional convolutions. For this, we adopt sparse tensors and propose the generalized sparse convolution that encompasses all discrete convolutions. To implement the generalized sparse convolution, we create an open-source auto-differentiation library for sparse tensors that provides extensive functions for high-dimensional convolutional neural networks. We create 4D spatio-temporal convolutional neural networks using the library and validate them on various 3D semantic segmentation benchmarks and proposed 4D datasets for 3D-video perception. To overcome challenges in the 4D space, we propose the hybrid kernel, a special case of the generalized sparse convolution, and the trilateral-stationary conditional random field that enforces spatio-temporal consistency in the 7D space-time-chroma space. Experimentally, we show that convolutional neural networks with only generalized 3D sparse convolutions can outperform 2D or 2D-3D hybrid methods by a large margin. Also, we show that on 3D-videos, 4D spatio-temporal convolutional neural networks are robust to noise, outperform 3D convolutional neural networks and are faster than the 3D counterpart in some cases.

Key facts. Cited by 1,201 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 152; 2022: 195; 2023: 226; 2024: 232; 2025: 241; 2026: 155.

1,201 papers.

Sort:

By area:

  1. 2022BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representationcited by 844
  2. 2023ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenescited by 747
  3. 2023ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoderscited by 703
  4. 2023Point Transformer V3: Simpler, Faster, Strongercited by 579
  5. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
  6. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 432
  7. 2023Tri-Perspective View for Vision-Based 3D Semantic Occupancy Predictioncited by 404
  8. 2022Point Transformer V2: Grouped Vector Attention and Partition-based Poolingcited by 384
  9. 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
  10. 2022HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interactioncited by 321
  11. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 299
  12. 2023OpenMask3D: Open-Vocabulary 3D Instance Segmentationcited by 288
38 more
  1. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  2. 2021PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector Representation for 3D Object Detectioncited by 232
  3. 2021Self-Supervised Pretraining of 3D Features on any Point-Cloudcited by 217
  4. 2022Omnivore: A Single Model for Many Visual Modalitiescited by 214
  5. 2021PAConv: Position Adaptive Convolution with Dynamic Kernel Assembling on Point Cloudscited by 211
  6. 2022SoftGroup for 3D Instance Segmentation on Point Cloudscited by 197
  7. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 195
  8. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 193
  9. 2023OpenShape: Scaling Up 3D Shape Representation Towards Open-World Understandingcited by 183
  10. 2021Spatio-temporal Self-Supervised Representation Learning for 3D Point Cloudscited by 180
  11. 2023CLIP2Scene: Towards Label-efficient 3D Scene Understanding by CLIPcited by 178
  12. 2022PLA: Language-Driven Open-Vocabulary 3D Scene Understandingcited by 176
  13. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  14. 2021RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentationcited by 173
  15. 2022Stratified Transformer for 3D Point Cloud Segmentationcited by 166
  16. 2021(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Networkcited by 164
  17. 2021A System for General In-Hand Object Re-Orientationcited by 155
  18. 2021CoFiNet: Reliable Coarse-to-fine Correspondences for Robust Point Cloud Registrationcited by 151
  19. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  20. 2021PointDSC: Robust Point Cloud Registration using Deep Spatial Consistencycited by 142
  21. 2022Focal Sparse Convolutional Networks for 3D Object Detectioncited by 142
  22. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  23. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 140
  24. 2023Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulationcited by 135
  25. 2023Robo3D: Towards Robust and Reliable 3D Perception against Corruptionscited by 128
  26. 2021Mix3D: Out-of-Context Data Augmentation for 3D Scenescited by 126
  27. 2022Image-to-Lidar Self-Supervised Distillation for Autonomous Driving Datacited by 126
  28. 2023Rethinking Range View Representation for LiDAR Segmentationcited by 126
  29. 2023Spherical Transformer for LiDAR-based 3D Recognitioncited by 124
  30. 20214DComplete: Non-Rigid Motion Estimation Beyond the Observable Surfacecited by 123
  31. 2023Segment Any Point Cloud Sequences by Distilling Vision Foundation Modelscited by 123
  32. 2022Delving into the Devils of Bird's-eye-view Perception: A Review, Evaluation and Recipecited by 122
  33. 2023DSVT: Dynamic Sparse Voxel Transformer with Rotated Setscited by 122
  34. 2021NeSF: Neural Semantic Fields for Generalizable Semantic Segmentation of 3D Scenescited by 119
  35. 2022Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentationcited by 118
  36. 2023PointGPT: Auto-regressively Generative Pre-training from Point Cloudscited by 115
  37. 2023Swin3D: A Pretrained Transformer Backbone for 3D Indoor Scene Understandingcited by 111
  38. 2022EfficientNeRF: Efficient Neural Radiance Fieldscited by 110

50 most cited of 1,201 All 1,201 (JSON)