# Recognizing Actions in Videos from Unseen Viewpoints

arXiv: https://arxiv.org/abs/2103.16516 · HTML: https://mlsanity.com/paper/2103.16516/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 46 references that are in the map.
- First posted on arXiv 30 Mar 2021.

## Builds on (most cited first, 26 of 26)

- [PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space](https://mlsanity.com/paper/1706.02413/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [ShapeNet: An Information-Rich 3D Model Repository](https://mlsanity.com/paper/1512.03012/) (2015)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [A Closer Look at Spatiotemporal Convolutions for Action Recognition](https://mlsanity.com/paper/1711.11248/) (2017)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [Spherical CNNs](https://mlsanity.com/paper/1801.10130/) (2018)
- [3D human pose estimation in video with temporal convolutions and semi-supervised training](https://mlsanity.com/paper/1811.11742/) (2018)
- [SynSin: End-to-end View Synthesis from a Single Image](https://mlsanity.com/paper/1912.08804/) (2019)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [Learning SO(3) Equivariant Representations with Spherical CNNs](https://mlsanity.com/paper/1711.06721/) (2017)
- [Camera Distance-aware Top-down Approach for 3D Multi-person Pose Estimation from a Single RGB Image](https://mlsanity.com/paper/1907.11346/) (2019)
- [Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos](https://mlsanity.com/paper/1804.09626/) (2018)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [3D Human Pose Estimation from a Single Image via Distance Matrix Regression](https://mlsanity.com/paper/1611.09010/) (2016)
- [Multi-view 3D Models from Single Images with a Convolutional Network](https://mlsanity.com/paper/1511.06702/) (2015)
- [Skeleton-based Action Recognition with Convolutional Neural Networks](https://mlsanity.com/paper/1704.07595/) (2017)
- [Equivariant Multi-View Networks](https://mlsanity.com/paper/1904.00993/) (2019)
- [Learning Spatiotemporal Features with 3D Convolutional Networks](https://mlsanity.com/paper/1412.0767/) (2014)
- [Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation](https://mlsanity.com/paper/1508.06708/) (2015)
- [Fine-grained Activity Recognition in Baseball Videos](https://mlsanity.com/paper/1804.03247/) (2018)
- [Geometry-Aware Recurrent Neural Networks for Active Visual Recognition](https://mlsanity.com/paper/1811.01292/) (2018)
- [Invariant recognition drives neural representations of action sequences](https://mlsanity.com/paper/1606.04698/) (2016)

## Built on by (most cited first, 18 of 18)

- [Learning Fine-grained View-Invariant Representations from Unpaired Ego-Exo Videos via Temporal Alignment](https://mlsanity.com/paper/2306.05526/) (2023)
- [UNIK: A Unified Framework for Real-world Skeleton-based Action Recognition](https://mlsanity.com/paper/2107.08580/) (2021)
- [ViA: View-invariant Skeleton Action Representation Learning via Motion Retargeting](https://mlsanity.com/paper/2209.00065/) (2022)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living](https://mlsanity.com/paper/2311.18840/) (2023)
- [DVANet: Disentangling View and Action Features for Multi-View Action Recognition](https://mlsanity.com/paper/2312.05719/) (2023)
- [Learning Viewpoint-Agnostic Visual Representations by Recovering Tokens in 3D Space](https://mlsanity.com/paper/2206.11895/) (2022)
- [Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos](https://mlsanity.com/paper/2203.13309/) (2022)
- [Self-Supervised Video Representation Learning via Latent Time Navigation](https://mlsanity.com/paper/2305.06437/) (2023)
- [POV: Prompt-Oriented View-Agnostic Learning for Egocentric Hand-Object Interaction in the Multi-View World](https://mlsanity.com/paper/2403.05856/) (2024)
- [MV2MAE: Multi-View Video Masked Autoencoders](https://mlsanity.com/paper/2401.15900/) (2024)
- [Transformers in Action Recognition: A Review on Temporal Modeling](https://mlsanity.com/paper/2302.01921/) (2022)
- [KAN-HyperpointNet for Point Cloud Sequence-Based 3D Human Action Recognition](https://mlsanity.com/paper/2409.09444/) (2024)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Cross-modal Manifold Cutmix for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03906/) (2021)
- [Towards Generalizing Temporal Action Segmentation to Unseen Views](https://mlsanity.com/paper/2504.02512/) (2025)
- [ViewBridge: Curriculum Knowledge Distillation for Activity View-Invariance Under Extreme Viewpoint Changes](https://mlsanity.com/paper/2504.05451/) (2025)
- [Probabilistic Temporal Masked Attention for Cross-view Online Action Detection](https://mlsanity.com/paper/2508.17025/) (2025)

## Abstract (arXiv)

Standard methods for video recognition use large CNNs designed to capture spatio-temporal data. However, training these models requires a large amount of labeled training data, containing a wide variety of actions, scenes, settings and camera viewpoints. In this paper, we show that current convolutional neural network models are unable to recognize actions from camera viewpoints not present in their training data (i.e., unseen view action recognition). To address this, we develop approaches based on 3D representations and introduce a new geometric convolutional layer that can learn viewpoint invariant representations. Further, we introduce a new, challenging dataset for unseen view recognition and show the approaches ability to learn viewpoint invariant representations.