# CaSPR: Learning Canonical Spatiotemporal Point Cloud Representations

arXiv: https://arxiv.org/abs/2008.02792 · HTML: https://mlsanity.com/paper/2008.02792/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Aug 2020.

## Built on by (most cited first, 30 of 51)

- [Neural Fields in Visual Computing and Beyond](https://mlsanity.com/paper/2111.11426/) (2021)
- [Neural Kernel Surface Reconstruction](https://mlsanity.com/paper/2305.19590/) (2023)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [Weakly Supervised Learning of Rigid 3D Scene Flow](https://mlsanity.com/paper/2102.08945/) (2021)
- [Neural Fields as Learnable Kernels for 3D Reconstruction](https://mlsanity.com/paper/2111.13674/) (2021)
- [CaDeX: Learning Canonical Deformation Coordinate Space for Dynamic Surface Representation via Neural Homeomorphism](https://mlsanity.com/paper/2203.16529/) (2022)
- [Beyond 3D Siamese Tracking: A Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2203.01730/) (2022)
- [Dynamic 3D Scene Analysis by Point Cloud Accumulation](https://mlsanity.com/paper/2207.12394/) (2022)
- [MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization](https://mlsanity.com/paper/2101.06605/) (2021)
- [ICP-Flow: LiDAR Scene Flow Estimation with ICP](https://mlsanity.com/paper/2402.17351/) (2024)
- [Quantum Permutation Synchronization](https://mlsanity.com/paper/2101.07755/) (2021)
- [Recent Trends in 3D Reconstruction of General Non-Rigid Scenes](https://mlsanity.com/paper/2403.15064/) (2024)
- [No Pain, Big Gain: Classify Dynamic Point Cloud Sequences with Static Models by Fitting Feature-level Space-time Surfaces](https://mlsanity.com/paper/2203.11113/) (2022)
- [Learning Compositional Representation for 4D Captures with Neural ODE](https://mlsanity.com/paper/2103.08271/) (2021)
- [Towards Efficient Graph Convolutional Networks for Point Cloud Handling](https://mlsanity.com/paper/2104.05706/) (2021)
- [PU-Flow: a Point Cloud Upsampling Network with Normalizing Flows](https://mlsanity.com/paper/2107.05893/) (2021)
- [Shape-Pose Disentanglement using SE(3)-equivariant Vector Neurons](https://mlsanity.com/paper/2204.01159/) (2022)
- [Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2308.09247/) (2023)
- [H4D: Human 4D Modeling by Learning Neural Compositional Representation](https://mlsanity.com/paper/2203.01247/) (2022)
- [An Effective Motion-Centric Paradigm for 3D Single Object Tracking in Point Clouds](https://mlsanity.com/paper/2303.12535/) (2023)
- [Point-Cloud Deep Learning of Porous Media for Permeability Prediction](https://mlsanity.com/paper/2107.14038/) (2021)
- [SpOT: Spatiotemporal Modeling for 3D Object Tracking](https://mlsanity.com/paper/2207.05856/) (2022)
- [FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow](https://mlsanity.com/paper/2307.08100/) (2023)
- [Learning Scene Dynamics from Point Cloud Sequences](https://mlsanity.com/paper/2111.08755/) (2021)
- [IDEA-Net: Dynamic 3D Point Cloud Interpolation via Deep Embedding Alignment](https://mlsanity.com/paper/2203.11590/) (2022)
- [Variational Inference for SDEs Driven by Fractional Noise](https://mlsanity.com/paper/2310.12975/) (2023)
- [Learning to Complete Object Shapes for Object-level Mapping in Dynamic Scenes](https://mlsanity.com/paper/2208.05067/) (2022)
- [LoRD: Local 4D Implicit Representation for High-Fidelity Dynamic Human Modeling](https://mlsanity.com/paper/2208.08622/) (2022)
- [DynoSurf: Neural Deformation-based Temporally Consistent Dynamic Surface Reconstruction](https://mlsanity.com/paper/2403.11586/) (2024)
- [Multi-body SE(3) Equivariance for Unsupervised Rigid Segmentation and Motion Estimation](https://mlsanity.com/paper/2306.05584/) (2023)

## Abstract (arXiv)

We propose CaSPR, a method to learn object-centric Canonical Spatiotemporal Point Cloud Representations of dynamically moving or evolving objects. Our goal is to enable information aggregation over time and the interrogation of object state at any spatiotemporal neighborhood in the past, observed or not. Different from previous work, CaSPR learns representations that support spacetime continuity, are robust to variable and irregularly spacetime-sampled point clouds, and generalize to unseen object instances. Our approach divides the problem into two subtasks. First, we explicitly encode time by mapping an input point cloud sequence to a spatiotemporally-canonicalized object space. We then leverage this canonicalization to learn a spatiotemporal latent representation using neural ordinary differential equations and a generative model of dynamically evolving shapes using continuous normalizing flows. We demonstrate the effectiveness of our method on several applications including shape reconstruction, camera pose estimation, continuous spatiotemporal sequence reconstruction, and correspondence estimation from irregularly or intermittently sampled observations.