# Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding

arXiv: https://arxiv.org/abs/2208.00281 · HTML: https://mlsanity.com/paper/2208.00281/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 44 references that are in the map.
- First posted on arXiv 30 Jul 2022.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [Big Transfer (BiT): General Visual Representation Learning](https://mlsanity.com/paper/1912.11370/) (2019)
- [PCT: Point cloud transformer](https://mlsanity.com/paper/2012.09688/) (2020)
- [Large-scale Point Cloud Semantic Segmentation with Superpoint Graphs](https://mlsanity.com/paper/1711.09869/) (2017)
- [HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object Interaction](https://mlsanity.com/paper/2203.01577/) (2022)
- [Submanifold Sparse Convolutional Networks](https://mlsanity.com/paper/1706.01307/) (2017)
- [Learning Shape Abstractions by Assembling Volumetric Primitives](https://mlsanity.com/paper/1612.00404/) (2016)
- [How to represent part-whole hierarchies in a neural network](https://mlsanity.com/paper/2102.12627/) (2021)
- [MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences](https://mlsanity.com/paper/1910.09165/) (2019)
- [Supervised Fitting of Geometric Primitives to 3D Point Clouds](https://mlsanity.com/paper/1811.08988/) (2018)
- [3D-PRNN: Generating Shape Primitives with Recurrent Neural Networks](https://mlsanity.com/paper/1708.01648/) (2017)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [A Universal Representation Transformer Layer for Few-Shot Image Classification](https://mlsanity.com/paper/2006.11702/) (2020)
- [VTNet: Visual Transformer Network for Object Goal Navigation](https://mlsanity.com/paper/2105.09447/) (2021)
- [3DV: 3D Dynamic Voxel for Action Recognition in Depth Video](https://mlsanity.com/paper/2005.05501/) (2020)
- [HPNet: Deep Primitive Segmentation Using Hybrid Representations](https://mlsanity.com/paper/2105.10620/) (2021)
- [Rank Pooling for Action Recognition](https://mlsanity.com/paper/1512.01848/) (2015)

## Built on by (most cited first, 30 of 35)

- [ShapeLLM: Universal 3D Object Understanding for Embodied Interaction](https://mlsanity.com/paper/2402.17766/) (2024)
- [An Outlook into the Future of Egocentric Vision](https://mlsanity.com/paper/2308.07123/) (2023)
- [Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning](https://mlsanity.com/paper/2212.05330/) (2022)
- [Interactive Humanoid: Online Full-Body Motion Reaction Synthesis with Social Affordance Canonicalization and Forecasting](https://mlsanity.com/paper/2312.08983/) (2023)
- [PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2305.04075/) (2023)
- [Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2308.09245/) (2023)
- [Align then Adapt: Rethinking Parameter-Efficient Transfer Learning in 4D Perception](https://mlsanity.com/paper/2602.23069/) (2026)
- [Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2308.09247/) (2023)
- [GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation, Demonstration, and Imitation](https://mlsanity.com/paper/2401.00929/) (2024)
- [X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2312.07378/) (2023)
- [VG4D: Vision-Language Model Goes 4D Video Recognition](https://mlsanity.com/paper/2404.11605/) (2024)
- [MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models](https://mlsanity.com/paper/2405.14338/) (2024)
- [Semantic Complete Scene Forecasting from a 4D Dynamic Point Cloud Sequence](https://mlsanity.com/paper/2312.08054/) (2023)
- [CrossVideo: Self-supervised Cross-modal Contrastive Learning for Point Cloud Video Understanding](https://mlsanity.com/paper/2401.09057/) (2024)
- [Diffusion Masked Pretraining for Dynamic Point Cloud](https://mlsanity.com/paper/2605.03639/) (2026)
- [Tri-Efficient Transfer Learning for Point Cloud Videos](https://mlsanity.com/paper/2606.24175/) (2026)
- [Contrastive Predictive Autoencoders for Dynamic Point Cloud Self-Supervised Learning](https://mlsanity.com/paper/2305.12959/) (2023)
- [NSM4D: Neural Scene Model Based Online 4D Point Cloud Sequence Understanding](https://mlsanity.com/paper/2310.08326/) (2023)
- [A Unified Framework for Human-centric Point Cloud Video Understanding](https://mlsanity.com/paper/2403.20031/) (2024)
- [Uni4D: A Unified Self-Supervised Learning Framework for Point Cloud Videos](https://mlsanity.com/paper/2504.04837/) (2025)
- [DPMix: Mixture of Depth and Point Cloud Video Experts for 4D Action Segmentation](https://mlsanity.com/paper/2307.16803/) (2023)
- [SPiKE: 3D Human Pose from Point Cloud Sequences](https://mlsanity.com/paper/2409.01879/) (2024)
- [Content-Distortion High-Order Interaction for Blind Image Quality Assessment](https://mlsanity.com/paper/2504.05076/) (2025)
- [Point Cloud Video Anomaly Detection Based on Point Spatio-Temporal Auto-Encoder](https://mlsanity.com/paper/2306.04466/) (2023)
- [Quality assessment of 3D human animation: Subjective and objective evaluation](https://mlsanity.com/paper/2505.23301/) (2025)
- [DeSPITE: Exploring Contrastive Deep Skeleton-Pointcloud-IMU-Text Embeddings for Advanced Point Cloud Human Activity Understanding](https://mlsanity.com/paper/2506.13897/) (2025)
- [PointNet4D: A Lightweight 4D Point Cloud Video Backbone for Online and Offline Perception in Robotic Applications](https://mlsanity.com/paper/2512.01383/) (2025)
- [STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding](https://mlsanity.com/paper/2604.11637/) (2026)
- [SRENet: Spectral Re-Entry Network for Point Cloud Action Recognition](https://mlsanity.com/paper/2606.03160/) (2026)
- [Cross4D-JEPA: Dense Cross-modal Correspondence Distillation for 4D Point Cloud Representation Learning](https://mlsanity.com/paper/2607.00514/) (2026)

## Abstract (arXiv)

This paper proposes a 4D backbone for long-term point cloud video understanding. A typical way to capture spatial-temporal context is using 4Dconv or transformer without hierarchy. However, those methods are neither effective nor efficient enough due to camera motion, scene changes, sampling patterns, and the complexity of 4D data. To address those issues, we leverage the primitive plane as a mid-level representation to capture the long-term spatial-temporal context in 4D point cloud videos and propose a novel hierarchical backbone named Point Primitive Transformer(PPTr), which is mainly composed of intra-primitive point transformers and primitive transformers. Extensive experiments show that PPTr outperforms the previous state of the arts on different tasks.