# Glimpse Clouds: Human Activity Recognition from Unstructured Feature Points

arXiv: https://arxiv.org/abs/1802.07898 · HTML: https://mlsanity.com/paper/1802.07898/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Feb 2018.

## Built on by (most cited first, 30 of 33)

- [AP-10K: A Benchmark for Animal Pose Estimation in the Wild](https://mlsanity.com/paper/2108.12617/) (2021)
- [APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking](https://mlsanity.com/paper/2206.05683/) (2022)
- [STAR-Transformer: A Spatio-temporal Cross Attention Transformer for Human Action Recognition](https://mlsanity.com/paper/2210.07503/) (2022)
- [VPN++: Rethinking Video-Pose embeddings for understanding Activities of Daily Living](https://mlsanity.com/paper/2105.08141/) (2021)
- [ViewCLR: Learning Self-supervised Video Representation for Unseen Viewpoints](https://mlsanity.com/paper/2112.03905/) (2021)
- [Cross-Modal Learning with 3D Deformable Attention for Action Recognition](https://mlsanity.com/paper/2212.05638/) (2022)
- [Just Add π! Pose Induced Video Transformers for Understanding Activities of Daily Living](https://mlsanity.com/paper/2311.18840/) (2023)
- [BM-NAS: Bilevel Multimodal Neural Architecture Search](https://mlsanity.com/paper/2104.09379/) (2021)
- [Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognition](https://mlsanity.com/paper/2110.14994/) (2021)
- [Skeleton-based Action Recognition via Temporal-Channel Aggregation](https://mlsanity.com/paper/2205.15936/) (2022)
- [From CNNs to Transformers in Multimodal Human Action Recognition: A Survey](https://mlsanity.com/paper/2405.15813/) (2024)
- [LLAVIDAL: A Large LAnguage VIsion Model for Daily Activities of Living](https://mlsanity.com/paper/2406.09390/) (2024)
- [Skeleton Sequence and RGB Frame Based Multi-Modality Feature Fusion Network for Action Recognition](https://mlsanity.com/paper/2202.11374/) (2022)
- [MV2MAE: Multi-View Video Masked Autoencoders](https://mlsanity.com/paper/2401.15900/) (2024)
- [Expressive Keypoints for Skeleton-based Action Recognition via Skeleton Transformation](https://mlsanity.com/paper/2406.18011/) (2024)
- [CZU-MHAD: A multimodal dataset for human action recognition utilizing a depth camera and 10 wearable inertial sensors](https://mlsanity.com/paper/2202.03283/) (2022)
- [Deep video representation learning: a survey](https://mlsanity.com/paper/2405.06574/) (2024)
- [Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2506.03440/) (2025)
- [Multi-Scale Semantics-Guided Neural Networks for Efficient Skeleton-Based Human Action Recognition](https://mlsanity.com/paper/2111.03993/) (2021)
- [Thales: Formulating and Estimating Architectural Vulnerability Factors for DNN Accelerators](https://mlsanity.com/paper/2212.02649/) (2022)
- [Extended multi-stream temporal-attention module for skeleton-based human action recognition (HAR)](https://mlsanity.com/paper/2411.06553/) (2024)
- [Content-Distortion High-Order Interaction for Blind Image Quality Assessment](https://mlsanity.com/paper/2504.05076/) (2025)
- [Nuisance-Label Supervision: Robustness Improvement by Free Labels](https://mlsanity.com/paper/2110.07118/) (2021)
- [GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction](https://mlsanity.com/paper/2210.13605/) (2022)
- [Cross-view Action Recognition via Contrastive View-invariant Representation](https://mlsanity.com/paper/2305.01733/) (2023)
- [Performance of different machine learning methods on activity recognition and pose estimation datasets](https://mlsanity.com/paper/2210.10247/) (2022)
- [2D Pose Estimation based Child Action Recognition](https://mlsanity.com/paper/2212.09027/) (2022)
- [Towards High Performance One-Stage Human Pose Estimation](https://mlsanity.com/paper/2301.04842/) (2023)
- [Seeing the Pose in the Pixels: Learning Pose-Aware Representations in Vision Transformers](https://mlsanity.com/paper/2306.09331/) (2023)
- [HRPVT: High-Resolution Pyramid Vision Transformer for medium and small-scale human pose estimation](https://mlsanity.com/paper/2410.22079/) (2024)

## Abstract (arXiv)

We propose a method for human activity recognition from RGB data that does not rely on any pose information during test time and does not explicitly calculate pose information internally. Instead, a visual attention module learns to predict glimpse sequences in each frame. These glimpses correspond to interest points in the scene that are relevant to the classified activities. No spatial coherence is forced on the glimpse locations, which gives the module liberty to explore different points at each frame and better optimize the process of scrutinizing visual information. Tracking and sequentially integrating this kind of unstructured data is a challenge, which we address by separating the set of glimpses from a set of recurrent tracking/recognition workers. These workers receive glimpses, jointly performing subsequent motion tracking and activity prediction. The glimpses are soft-assigned to the workers, optimizing coherence of the assignments in space, time and feature space using an external memory module. No hard decisions are taken, i.e. each glimpse point is assigned to all existing workers, albeit with different importance. Our methods outperform state-of-the-art methods on the largest human activity recognition dataset available to-date; NTU RGB+D Dataset, and on a smaller human action recognition dataset Northwestern-UCLA Multiview Action 3D Dataset. Our code is publicly available at https://github.com/fabienbaradel/glimpse_clouds.