# Open Set Action Recognition via Multi-Label Evidential Learning

arXiv: https://arxiv.org/abs/2303.12698 · HTML: https://mlsanity.com/paper/2303.12698/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 54 references that are in the map.
- First posted on arXiv 27 Feb 2023.

## Builds on (most cited first, 30 of 42)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Energy-based Out-of-distribution Detection](https://mlsanity.com/paper/2010.03759/) (2020)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [X3D: Expanding Architectures for Efficient Video Recognition](https://mlsanity.com/paper/2004.04730/) (2020)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding](https://mlsanity.com/paper/1604.01753/) (2016)
- [AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actions](https://mlsanity.com/paper/1705.08421/) (2017)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Video Classification with Channel-Separated Convolutional Networks](https://mlsanity.com/paper/1904.02811/) (2019)
- [Long-Term Feature Banks for Detailed Video Understanding](https://mlsanity.com/paper/1812.05038/) (2018)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Large-scale weakly-supervised pre-training for video action recognition](https://mlsanity.com/paper/1905.00561/) (2019)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [Can multi-label classification networks know what they don't know?](https://mlsanity.com/paper/2109.14162/) (2021)
- [Evidential Deep Learning for Open Set Action Recognition](https://mlsanity.com/paper/2107.10161/) (2021)
- [Actor-Context-Actor Relation Network for Spatio-Temporal Action Localization](https://mlsanity.com/paper/2006.07976/) (2020)
- [Actor-Centric Relation Network](https://mlsanity.com/paper/1807.10982/) (2018)

## Built on by (most cited first, 26 of 26)

- [A Comprehensive Survey on Evidential Deep Learning and Its Applications](https://mlsanity.com/paper/2409.04720/) (2024)
- [Opening the Vocabulary of Egocentric Actions](https://mlsanity.com/paper/2308.11488/) (2023)
- [Revisiting Essential and Nonessential Settings of Evidential Deep Learning](https://mlsanity.com/paper/2410.00393/) (2024)
- [Uncertainty-Guided Appearance-Motion Association Network for Out-of-Distribution Action Detection](https://mlsanity.com/paper/2409.09953/) (2024)
- [Advancing Open-Set Domain Generalization Using Evidential Bi-Level Hardest Domain Scheduler](https://mlsanity.com/paper/2409.17555/) (2024)
- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [Mitigating Label Noise using Prompt-Based Hyperbolic Meta-Learning in Open-Set Domain Generalization](https://mlsanity.com/paper/2412.18342/) (2024)
- [Uncertainty Estimation for 3D Object Detection via Evidential Learning](https://mlsanity.com/paper/2410.23910/) (2024)
- [MADOD: Generalizing OOD Detection to Unseen Domains via G-Invariance Meta-Learning](https://mlsanity.com/paper/2411.02444/) (2024)
- [Managing the unknown: a survey on Open Set Recognition and tangential areas](https://mlsanity.com/paper/2312.08785/) (2023)
- [Weakly-Supervised Residual Evidential Learning for Multi-Instance Uncertainty Estimation](https://mlsanity.com/paper/2405.04405/) (2024)
- [Situational Scene Graph for Structured Human-centric Situation Understanding](https://mlsanity.com/paper/2410.22829/) (2024)
- [Towards Effective Semantic OOD Detection in Unseen Domains: A Domain Generalization Perspective](https://mlsanity.com/paper/2309.10209/) (2023)
- [Skeleton-OOD: An End-to-End Skeleton-Based Model for Robust Out-of-Distribution Human Action Detection](https://mlsanity.com/paper/2405.20633/) (2024)
- [Revisiting Uncertainty: On Evidential Learning for Partially Relevant Video Retrieval](https://mlsanity.com/paper/2605.06083/) (2026)
- [Cross Domain Object Detection via Multi-Granularity Confidence Alignment based Mean Teacher](https://mlsanity.com/paper/2407.07780/) (2024)
- [Feature-Space Semantic Invariance: Enhanced OOD Detection for Open-Set Domain Generalization](https://mlsanity.com/paper/2411.07392/) (2024)
- [Markov Process-Based Graph Convolutional Networks for Entity Classification in Knowledge Graphs](https://mlsanity.com/paper/2412.17438/) (2024)
- [Multi-label out-of-distribution detection via evidential learning](https://mlsanity.com/paper/2502.18224/) (2025)
- [EReLiFM: Evidential Reliability-Aware Residual Flow Meta-Learning for Open-Set Domain Generalization under Noisy Labels](https://mlsanity.com/paper/2510.12687/) (2025)
- [The impact of Compositionality in Zero-shot Multi-label action recognition for Object-based tasks](https://mlsanity.com/paper/2405.08695/) (2024)
- [Reliable Imputed-Sample Assisted Vertical Federated Learning](https://mlsanity.com/paper/2501.06429/) (2025)
- [Out-of-Distribution Detection in Heterogeneous Graphs via Energy Propagation](https://mlsanity.com/paper/2505.03774/) (2025)
- [EVINET: Towards Open-World Graph Learning via Evidential Reasoning Network](https://mlsanity.com/paper/2506.07288/) (2025)
- [Open Set Face Forgery Detection via Dual-Level Evidence Collection](https://mlsanity.com/paper/2512.04331/) (2025)
- [UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection](https://mlsanity.com/paper/2604.21227/) (2026)

## Abstract (arXiv)

Existing methods for open-set action recognition focus on novelty detection that assumes video clips show a single action, which is unrealistic in the real world. We propose a new method for open set action recognition and novelty detection via MUlti-Label Evidential learning (MULE), that goes beyond previous novel action detection methods by addressing the more general problems of single or multiple actors in the same scene, with simultaneous action(s) by any actor. Our Beta Evidential Neural Network estimates multi-action uncertainty with Beta densities based on actor-context-object relation representations. An evidence debiasing constraint is added to the objective function for optimization to reduce the static bias of video representations, which can incorrectly correlate predictions and static cues. We develop a learning algorithm based on a primal-dual average scheme update to optimize the proposed problem. Theoretical analysis of the optimization algorithm demonstrates the convergence of the primal solution sequence and bounds for both the loss function and the debiasing constraint. Uncertainty and belief-based novelty estimation mechanisms are formulated to detect novel actions. Extensive experiments on two real-world video datasets show that our proposed approach achieves promising performance in single/multi-actor, single/multi-action settings.