# Navigating Open Set Scenarios for Skeleton-based Action Recognition

arXiv: https://arxiv.org/abs/2312.06330 · HTML: https://mlsanity.com/paper/2312.06330/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 11 Dec 2023.

## Builds on (most cited first, 20 of 20)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [Channel-wise Topology Refinement Graph Convolution for Skeleton-Based Action Recognition](https://mlsanity.com/paper/2107.12213/) (2021)
- [Classification-Reconstruction Learning for Open-Set Recognition](https://mlsanity.com/paper/1812.04246/) (2018)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [A New Representation of Skeleton Sequences for 3D Action Recognition](https://mlsanity.com/paper/1703.03492/) (2017)
- [Learning Open Set Network with Discriminative Reciprocal Points](https://mlsanity.com/paper/2011.00178/) (2020)
- [Dropout Sampling for Robust Object Detection in Open-Set Conditions](https://mlsanity.com/paper/1710.06677/) (2017)
- [Conditional Gaussian Distribution Learning for Open Set Recognition](https://mlsanity.com/paper/2003.08823/) (2020)
- [Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition](https://mlsanity.com/paper/2007.14690/) (2020)
- [Spatial Temporal Transformer Network for Skeleton-based Action Recognition](https://mlsanity.com/paper/2012.06399/) (2020)
- [Toyota Smarthome Untrimmed: Real-World Untrimmed Videos for Activity Detection](https://mlsanity.com/paper/2010.14982/) (2020)
- [PMAL: Open Set Recognition via Robust Prototype Mining](https://mlsanity.com/paper/2203.08569/) (2022)
- [ODN: Opening the Deep Network for Open-set Action Recognition](https://mlsanity.com/paper/1901.07757/) (2019)
- [BAR: Bayesian Activity Recognition using variational inference](https://mlsanity.com/paper/1811.03305/) (2018)
- [HyperFormer: Learning Expressive Sparse Feature Representations via Hypergraph Transformer](https://mlsanity.com/paper/2305.17386/) (2023)
- [Reconstructing Humpty Dumpty: Multi-feature Graph Autoencoder for Open Set Action Recognition](https://mlsanity.com/paper/2212.06023/) (2022)

## Built on by (most cited first, 17 of 17)

- [Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler](https://mlsanity.com/paper/2409.17555/) (2024)
- [Skeleton-Based Human Action Recognition with Noisy Labels](https://mlsanity.com/paper/2403.09975/) (2024)
- [Mitigating Label Noise using Prompt-Based Hyperbolic Meta-Learning in Open-Set Domain Generalization](https://mlsanity.com/paper/2412.18342/) (2024)
- [Driver Activity Classification Using Generalizable Representations from Vision-Language Models](https://mlsanity.com/paper/2404.14906/) (2024)
- [CHASE: Learning Convex Hull Adaptive Shift for Skeleton-based Multi-Entity Action Recognition](https://mlsanity.com/paper/2410.07153/) (2024)
- [Muscles in Time: Learning to Understand Human Motion by Simulating Muscle Activations](https://mlsanity.com/paper/2411.00128/) (2024)
- [ProbRes: Probabilistic Jump Diffusion for Open-World Egocentric Activity Recognition](https://mlsanity.com/paper/2504.03948/) (2025)
- [IMPACT-CYCLE: A Contract-Based Multi-Agent System for Claim-Level Supervisory Correction of Long-Video Semantic Memory](https://mlsanity.com/paper/2604.20136/) (2026)
- [IMPACT-HOI: Supervisory Control for Onset-Anchored Partial HOI Event Construction](https://mlsanity.com/paper/2605.01666/) (2026)
- [Exploring Video-Based Driver Activity Recognition under Noisy Labels](https://mlsanity.com/paper/2504.11966/) (2025)
- [EReLiFM: Evidential Reliability-Aware Residual Flow Meta-Learning for Open-Set Domain Generalization under Noisy Labels](https://mlsanity.com/paper/2510.12687/) (2025)
- [PROVIA: Procedure State Tracking for Online Mistake Detection in Egocentric Videos](https://mlsanity.com/paper/2609.20638/) (2026)
- [PovNet+: A Deep Learning Architecture for Socially Assistive Robots to Learn and Assist with Multiple Activities of Daily Living](https://mlsanity.com/paper/2602.00131/) (2026)
- [A Baseline Study and Benchmark for Few-Shot Open-Set Action Recognition with Feature Residual Discrimination](https://mlsanity.com/paper/2603.04125/) (2026)
- [SkelHCC: A Hyperbolic CLIP-Driven Cache Adaptation Framework for Skeleton-based One-Shot Action Recognition](https://mlsanity.com/paper/2606.03610/) (2026)
- [STAR: Skeletal Token Alignment and Rearrangement for Interaction Recognition](https://mlsanity.com/paper/2607.17342/) (2026)
- [De-biasing Skeleton-based Action Recognition with Convex Hull Adaptive Shift](https://mlsanity.com/paper/2609.32454/) (2026)

## Abstract (arXiv)

In real-world scenarios, human actions often fall outside the distribution of training data, making it crucial for models to recognize known actions and reject unknown ones. However, using pure skeleton data in such open-set conditions poses challenges due to the lack of visual background cues and the distinct sparse structure of body pose sequences. In this paper, we tackle the unexplored Open-Set Skeleton-based Action Recognition (OS-SAR) task and formalize the benchmark on three skeleton-based datasets. We assess the performance of seven established open-set approaches on our task and identify their limits and critical generalization issues when dealing with skeleton information. To address these challenges, we propose a distance-based cross-modality ensemble method that leverages the cross-modal alignment of skeleton joints, bones, and velocities to achieve superior open-set recognition performance. We refer to the key idea as CrossMax - an approach that utilizes a novel cross-modality mean max discrepancy suppression mechanism to align latent spaces during training and a cross-modality distance-based logits refinement method during testing. CrossMax outperforms existing approaches and consistently yields state-of-the-art results across all datasets and backbones. The benchmark, code, and models will be released at https://github.com/KPeng9510/OS-SAR.