# Hierarchically Learned View-Invariant Representations for Cross-View Action Recognition

arXiv: https://arxiv.org/abs/1809.00421 · HTML: https://mlsanity.com/paper/1809.00421/

## Key facts

- Cited by 16 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 3 Sep 2018.

## Built on by (most cited first, 16 of 16)

- [SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decomposition](https://mlsanity.com/paper/2106.03849/) (2021)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [SkeletonMAE: Graph-based Masked Autoencoder for Skeleton Sequence Pre-training](https://mlsanity.com/paper/2307.08476/) (2023)
- [TCGL: Temporal Contrastive Graph for Self-supervised Video Representation Learning](https://mlsanity.com/paper/2112.03587/) (2021)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Visual Causal Scene Refinement for Video Question Answering](https://mlsanity.com/paper/2305.04224/) (2023)
- [Urban Regional Function Guided Traffic Flow Prediction](https://mlsanity.com/paper/2303.09789/) (2023)
- [DenseLight: Efficient Control for Large-scale Traffic Signals with Dense Feedback](https://mlsanity.com/paper/2306.07553/) (2023)
- [Temporal Contrastive Graph Learning for Video Action Recognition and Retrieval](https://mlsanity.com/paper/2101.00820/) (2021)
- [CausalVLR: A Toolbox and Benchmark for Visual-Linguistic Causal Reasoning](https://mlsanity.com/paper/2306.17462/) (2023)
- [Self-supervised Contrastive Learning for Audio-Visual Action Recognition](https://mlsanity.com/paper/2204.13386/) (2022)
- [MEIA: Multimodal Embodied Perception and Interaction in Unknown Environments](https://mlsanity.com/paper/2402.00290/) (2024)
- [VCD: Visual Causality Discovery for Cross-Modal Question Reasoning](https://mlsanity.com/paper/2304.08083/) (2023)
- [Cross-view Action Recognition via Contrastive View-invariant Representation](https://mlsanity.com/paper/2305.01733/) (2023)
- [Multi Camera Connected Vision System with Multi View Analytics: A Comprehensive Survey](https://mlsanity.com/paper/2510.09731/) (2025)
- [Context-Aware Network Based on Multi-scale Spatio-temporal Attention for Action Recognition in Videos](https://mlsanity.com/paper/2512.18750/) (2025)

## Abstract (arXiv)

Recognizing human actions from varied views is challenging due to huge appearance variations in different views. The key to this problem is to learn discriminant view-invariant representations generalizing well across views. In this paper, we address this problem by learning view-invariant representations hierarchically using a novel method, referred to as Joint Sparse Representation and Distribution Adaptation (JSRDA). To obtain robust and informative feature representations, we first incorporate a sample-affinity matrix into the marginalized stacked denoising Autoencoder (mSDA) to obtain shared features, which are then combined with the private features. In order to make the feature representations of videos across views transferable, we then learn a transferable dictionary pair simultaneously from pairs of videos taken at different views to encourage each action video across views to have the same sparse representation. However, the distribution difference across views still exists because a unified subspace where the sparse representations of one action across views are the same may not exist when the view difference is large. Therefore, we propose a novel unsupervised distribution adaptation method that learns a set of projections that project the source and target views data into respective low-dimensional subspaces where the marginal and conditional distribution differences are reduced simultaneously. Therefore, the finally learned feature representation is view-invariant and robust for substantial distribution difference across views even the view difference is large. Experimental results on four multiview datasets show that our approach outperforms the state-ofthe-art approaches.