# STS-Mixer: Spatio-Temporal-Spectral Mixer for 4D Point Cloud Video Understanding

arXiv: https://arxiv.org/abs/2604.11637 · HTML: https://mlsanity.com/paper/2604.11637/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 48 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Builds on (most cited first, 30 of 33)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks](https://mlsanity.com/paper/1904.08755/) (2019)
- [Rethinking Network Design and Local Geometry in Point Cloud: A Simple Residual MLP Framework](https://mlsanity.com/paper/2202.07123/) (2022)
- [PointMamba: A Simple State Space Model for Point Cloud Analysis](https://mlsanity.com/paper/2402.10739/) (2024)
- [Modeling Point Clouds with Self-Attention and Gumbel Subset Sampling](https://mlsanity.com/paper/1904.03375/) (2019)
- [MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation](https://mlsanity.com/paper/2111.12707/) (2021)
- [MeteorNet: Deep Learning on Dynamic 3D Point Cloud Sequences](https://mlsanity.com/paper/1910.09165/) (2019)
- [PointMixer: MLP-Mixer for Point Cloud Understanding](https://mlsanity.com/paper/2111.11187/) (2021)
- [PSTNet: Point Spatio-Temporal Convolution on Point Cloud Sequences](https://mlsanity.com/paper/2205.13713/) (2022)
- [Point Cloud Mamba: Point Cloud Learning via State Space Model](https://mlsanity.com/paper/2403.00762/) (2024)
- [Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning](https://mlsanity.com/paper/2410.08114/) (2024)
- [Point Primitive Transformer for Long-Term 4D Point Cloud Video Understanding](https://mlsanity.com/paper/2208.00281/) (2022)
- [PointRNN: Point Recurrent Neural Network for Moving Point Cloud Processing](https://mlsanity.com/paper/1910.08287/) (2019)
- [Complete-to-Partial 4D Distillation for Self-Supervised Point Cloud Sequence Representation Learning](https://mlsanity.com/paper/2212.05330/) (2022)
- [Fast Resampling of 3D Point Clouds via Graphs](https://mlsanity.com/paper/1702.06397/) (2017)
- [Graph Signal Processing for Geometric Data and Beyond: Theory and Applications](https://mlsanity.com/paper/2008.01918/) (2020)
- [PointCMP: Contrastive Mask Prediction for Self-supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2305.04075/) (2023)
- [Exploring the Devil in Graph Spectral Domain for 3D Point Cloud Attacks](https://mlsanity.com/paper/2202.07261/) (2022)
- [Masked Spatio-Temporal Structure Prediction for Self-supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2308.09245/) (2023)
- [Spectral-GANs for High-Resolution 3D Point-cloud Generation](https://mlsanity.com/paper/1912.01800/) (2019)
- [Point Cloud Attacks in Graph Spectral Domain: When 3D Geometry Meets Graph Signal Processing](https://mlsanity.com/paper/2207.13326/) (2022)
- [Point Contrastive Prediction with Semantic Clustering for Self-Supervised Learning on Point Cloud Videos](https://mlsanity.com/paper/2308.09247/) (2023)
- [Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation](https://mlsanity.com/paper/2311.12028/) (2023)
- [Hypergraph Spectral Analysis and Processing in 3D Point Cloud](https://mlsanity.com/paper/2001.02384/) (2020)
- [X4D-SceneFormer: Enhanced Scene Understanding on 4D Point Cloud Videos through Cross-modal Knowledge Transfer](https://mlsanity.com/paper/2312.07378/) (2023)
- [MAMBA4D: Efficient Long-Sequence Point Cloud Video Understanding with Disentangled Spatial-Temporal State Space Models](https://mlsanity.com/paper/2405.14338/) (2024)

## Built on by (most cited first, 1 of 1)

- [Joint-Embedding Prediction of Masked Point Tubes for Self-Supervised Learning on 4D Point Cloud Videos](https://mlsanity.com/paper/2608.24093/) (2026)

## Abstract (arXiv)

4D point cloud videos capture rich spatial and temporal dynamics of scenes which possess unique values in various 4D understanding tasks. However, most existing methods work in the spatiotemporal domain where the underlying geometric characteristics of 4D point cloud videos are hard to capture, leading to degraded representation learning and understanding of 4D point cloud videos. We address the above challenge from a complementary spectral perspective. By transforming 4D point cloud videos into graph spectral signals, we can decompose them into multiple frequency bands each of which captures distinct geometric structures of point cloud videos. Our spectral analysis reveals that the decomposed low-frequency signals capture more coarse shapes while high-frequency signals encode more fine-grained geometry details. Building on these observations, we design Spatio-Temporal-Spectral Mixer (STS-Mixer), a unified framework that mixes spatial, temporal, and spectral representations of point cloud videos. STS-Mixer integrates multi-band delineated spectral signals with spatiotemporal information to capture rich geometries and temporal dynamics, while enabling fine-grained and holistic understanding of 4D point cloud videos. Extensive experiments show that STS-Mixer achieves superior performance consistently across multiple widely adopted benchmarks on both 3D action recognition and 4D semantic segmentation tasks. Code and models are available at https://github.com/Vegetebird/STS-Mixer.