# DVANet: Disentangling View and Action Features for Multi-View Action Recognition

arXiv: https://arxiv.org/abs/2312.05719 · HTML: https://mlsanity.com/paper/2312.05719/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 48 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Builds on (most cited first, 25 of 25)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Disentangling and Unifying Graph Convolutions for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2003.14111/) (2020)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Towards a Definition of Disentangled Representations](https://mlsanity.com/paper/1812.02230/) (2018)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Joint Discriminative and Generative Learning for Person Re-identification](https://mlsanity.com/paper/1904.07223/) (2019)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data](https://mlsanity.com/paper/1611.06067/) (2016)
- [Co-occurrence Feature Learning from Skeleton Data for Action Recognition and Detection with Hierarchical Aggregation](https://mlsanity.com/paper/1804.06055/) (2018)
- [PKU-MMD: A Large Scale Benchmark for Continuous Multi-Modal Human Action Understanding](https://mlsanity.com/paper/1703.07475/) (2017)
- [Contrastive Learning from Extremely Augmented Skeleton Sequences for Self-supervised Action Recognition](https://mlsanity.com/paper/2112.03590/) (2021)
- [PYSKL: Towards Good Practices for Skeleton Action Recognition](https://mlsanity.com/paper/2205.09443/) (2022)
- [Gait Recognition via Disentangled Representation Learning](https://mlsanity.com/paper/1904.04925/) (2019)
- [Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization](https://mlsanity.com/paper/2103.15537/) (2021)
- [VPN: Learning Video-Pose Embedding for Activities of Daily Living](https://mlsanity.com/paper/2007.03056/) (2020)
- [Human-Object Interaction Detection via Disentangled Transformer](https://mlsanity.com/paper/2204.09290/) (2022)
- [Unsupervised Learning of View-invariant Action Representations](https://mlsanity.com/paper/1809.01844/) (2018)
- [Hierarchical Consistent Contrastive Learning for Skeleton-Based Action Recognition with Growing Augmentations](https://mlsanity.com/paper/2211.13466/) (2022)
- [AdaSGN: Adapting Joint Number and Model Size for Efficient Skeleton-Based Action Recognition](https://mlsanity.com/paper/2103.11770/) (2021)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [Learning View-Disentangled Human Pose Representation by Contrastive Cross-View Mutual Information Maximization](https://mlsanity.com/paper/2012.01405/) (2020)
- [Recognizing Actions in Videos from Unseen Viewpoints](https://mlsanity.com/paper/2103.16516/) (2021)
- [TSM: Temporal Shift Module for Efficient and Scalable Video Understanding on Edge Device](https://mlsanity.com/paper/2109.13227/) (2021)
- [Decoupling and Recoupling Spatiotemporal Representation for RGB-D-based Motion Recognition](https://mlsanity.com/paper/2112.09129/) (2021)

## Built on by (most cited first, 19 of 19)

- [SVFormer: A Direct Training Spiking Transformer for Efficient Video Action Recognition](https://mlsanity.com/paper/2406.15034/) (2024)
- [Probing Fine-Grained Action Understanding and Cross-View Generalization of Foundation Models](https://mlsanity.com/paper/2407.15605/) (2024)
- [CLAD: Constrained Latent Action Diffusion for Vision-Language Procedure Planning](https://mlsanity.com/paper/2503.06637/) (2025)
- [Joint-Dataset Learning and Cross-Consistent Regularization for Text-to-Motion Retrieval](https://mlsanity.com/paper/2407.02104/) (2024)
- [Cross-Camera Distracted Driver Classification through Feature Disentanglement and Contrastive Learning](https://mlsanity.com/paper/2411.13181/) (2024)
- [Trunk-branch Contrastive Network with Multi-view Deformable Aggregation for Multi-view Action Recognition](https://mlsanity.com/paper/2502.16493/) (2025)
- [Beyond the Horizon: Decoupling Multi-View UAV Action Recognition via Partial Order Transfer](https://mlsanity.com/paper/2504.20530/) (2025)
- [An LLM-Empowered Low-Resolution Vision System for On-Device Human Behavior Understanding](https://mlsanity.com/paper/2505.01743/) (2025)
- [Punching Bag vs. Punching Person: Motion Transferability in Videos](https://mlsanity.com/paper/2508.00085/) (2025)
- [Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Survey](https://mlsanity.com/paper/2510.09731/) (2025)
- [Heatmap Pooling Network for Action Recognition from RGB Videos](https://mlsanity.com/paper/2512.03837/) (2025)
- [LS-HAR: Language Supervised Human Action Recognition with Salient Fusion, Construction Sites as a Use-Case](https://mlsanity.com/paper/2410.01962/) (2024)
- [Towards Generalizing Temporal Action Segmentation to Unseen Views](https://mlsanity.com/paper/2504.02512/) (2025)
- [Probabilistic Temporal Masked Attention for Cross-view Online Action Detection](https://mlsanity.com/paper/2508.17025/) (2025)
- [StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales](https://mlsanity.com/paper/2510.16209/) (2025)
- [Beyond Real versus Fake Towards Intent-Aware Video Analysis](https://mlsanity.com/paper/2511.22455/) (2025)
- [Resource-Efficient RGB-Only Action Recognition for Edge Deployment](https://mlsanity.com/paper/2602.10818/) (2026)
- [SHANDS: A Multi-View Dataset and Benchmark for Surgical Hand-Gesture and Error Recognition Toward Medical Training](https://mlsanity.com/paper/2603.26400/) (2026)
- [Cross-Domain Human Action Recognition from Multiview Motion and Textual Descriptions](https://mlsanity.com/paper/2605.22697/) (2026)

## Abstract (arXiv)

In this work, we present a novel approach to multi-view action recognition where we guide learned action representations to be separated from view-relevant information in a video. When trying to classify action instances captured from multiple viewpoints, there is a higher degree of difficulty due to the difference in background, occlusion, and visibility of the captured action from different camera angles. To tackle the various problems introduced in multi-view action recognition, we propose a novel configuration of learnable transformer decoder queries, in conjunction with two supervised contrastive losses, to enforce the learning of action features that are robust to shifts in viewpoints. Our disentangled feature learning occurs in two stages: the transformer decoder uses separate queries to separately learn action and view information, which are then further disentangled using our two contrastive losses. We show that our model and method of training significantly outperforms all other uni-modal models on four multi-view action recognition datasets: NTU RGB+D, NTU RGB+D 120, PKU-MMD, and N-UCLA. Compared to previous RGB works, we see maximal improvements of 1.5%, 4.8%, 2.2%, and 4.8% on each dataset, respectively.