# Evidential Deep Learning for Open Set Action Recognition

arXiv: https://arxiv.org/abs/2107.10161 · HTML: https://mlsanity.com/paper/2107.10161/

## Key facts

- Cited by 96 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 67 references that are in the map.
- First posted on arXiv 21 Jul 2021.

## Builds on (most cited first, 30 of 37)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [SphereFace: Deep Hypersphere Embedding for Face Recognition](https://mlsanity.com/paper/1704.08063/) (2017)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Counts](https://mlsanity.com/paper/2006.09239/) (2020)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [Evaluating Bayesian Deep Learning Methods for Semantic Segmentation](https://mlsanity.com/paper/1811.12709/) (2018)
- [C2AE: Class Conditioned Auto-Encoder for Open-set Recognition](https://mlsanity.com/paper/1904.01198/) (2019)
- [Improving model calibration with accuracy versus uncertainty optimization](https://mlsanity.com/paper/2012.07923/) (2020)
- [Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition](https://mlsanity.com/paper/1912.05534/) (2019)
- [Learning Open Set Network with Discriminative Reciprocal Points](https://mlsanity.com/paper/2011.00178/) (2020)
- [Conditional Gaussian Distribution Learning for Open Set Recognition](https://mlsanity.com/paper/2003.08823/) (2020)
- [Uncertainty-based Traffic Accident Anticipation with Spatio-Temporal Relational Learning](https://mlsanity.com/paper/2008.00334/) (2020)
- [Out-of-Distribution Detection for Generalized Zero-Shot Action Recognition](https://mlsanity.com/paper/1904.08703/) (2019)
- [Uncertainty-Aware Deep Classifiers using Generative Models](https://mlsanity.com/paper/2006.04183/) (2020)
- [Open-world Learning and Application to Product Classification](https://mlsanity.com/paper/1809.06004/) (2018)
- [Multi-Moments in Time: Learning and Interpreting Models for Multi-Action Video Understanding](https://mlsanity.com/paper/1911.00232/) (2019)

## Built on by (most cited first, 30 of 96)

- [InternVideo: General Video Foundation Models via Generative and Discriminative Learning](https://mlsanity.com/paper/2212.03191/) (2022)
- [Trusted Multi-View Classification with Dynamic Evidential Fusion](https://mlsanity.com/paper/2204.11423/) (2022)
- [Is Out-of-Distribution Detection Learnable?](https://mlsanity.com/paper/2210.14707/) (2022)
- [UBnormal: New Benchmark for Supervised Open-Set Video Anomaly Detection](https://mlsanity.com/paper/2111.08644/) (2021)
- [A Survey on Uncertainty Quantification Methods for Deep Learning](https://mlsanity.com/paper/2302.13425/) (2023)
- [Prior and Posterior Networks: A Survey on Evidential Deep Learning Methods For Uncertainty Estimation](https://mlsanity.com/paper/2110.03051/) (2021)
- [Nearest Neighbor Guidance for Out-of-Distribution Detection](https://mlsanity.com/paper/2309.14888/) (2023)
- [Uncertainty Estimation by Fisher Information-based Evidential Deep Learning](https://mlsanity.com/paper/2303.02045/) (2023)
- [Benchmarking Micro-action Recognition: Dataset, Methods, and Applications](https://mlsanity.com/paper/2403.05234/) (2024)
- [On Second-Order Scoring Rules for Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2301.12736/) (2023)
- [Is Epistemic Uncertainty Faithfully Represented by Evidential Deep Learning Methods?](https://mlsanity.com/paper/2402.09056/) (2024)
- [Learn to Accumulate Evidence from All Training Samples: Theory and Practice](https://mlsanity.com/paper/2306.11113/) (2023)
- [Towards Open Set Video Anomaly Detection](https://mlsanity.com/paper/2208.11113/) (2022)
- [Open Set Action Recognition via Multi-Label Evidential Learning](https://mlsanity.com/paper/2303.12698/) (2023)
- [A Comprehensive Survey on Evidential Deep Learning and Its Applications](https://mlsanity.com/paper/2409.04720/) (2024)
- [Causal Reasoning Meets Visual Representation Learning: A Prospective Study](https://mlsanity.com/paper/2204.12037/) (2022)
- [Dirichlet-based Uncertainty Calibration for Active Domain Adaptation](https://mlsanity.com/paper/2302.13824/) (2023)
- [OpenTAL: Towards Open Set Temporal Action Localization](https://mlsanity.com/paper/2203.05114/) (2022)
- [Opening the Vocabulary of Egocentric Actions](https://mlsanity.com/paper/2308.11488/) (2023)
- [MADiff: Motion-Aware Mamba Diffusion Models for Hand Trajectory Prediction on Egocentric Videos](https://mlsanity.com/paper/2409.02638/) (2024)
- [The Devil is in the Wrongly-classified Samples: Towards Unified Open-set Recognition](https://mlsanity.com/paper/2302.04002/) (2023)
- [EdVAE: Mitigating Codebook Collapse with Evidential Discrete Variational Autoencoders](https://mlsanity.com/paper/2310.05718/) (2023)
- [Evidential Active Recognition: Intelligent and Prudent Open-World Embodied Perception](https://mlsanity.com/paper/2311.13793/) (2023)
- [Revisiting Essential and Nonessential Settings of Evidential Deep Learning](https://mlsanity.com/paper/2410.00393/) (2024)
- [Evidential Detection and Tracking Collaboration: New Problem, Benchmark and Algorithm for Robust Anti-UAV System](https://mlsanity.com/paper/2306.15767/) (2023)
- [Evidential Uncertainty Quantification: A Variance-Based Perspective](https://mlsanity.com/paper/2311.11367/) (2023)
- [Open-Set Recognition in the Age of Vision-Language Models](https://mlsanity.com/paper/2403.16528/) (2024)
- [SOAR: Scene-debiasing Open-set Action Recognition](https://mlsanity.com/paper/2309.01265/) (2023)
- [CREST: Cross-modal Resonance through Evidential Deep Learning for Enhanced Zero-Shot Learning](https://mlsanity.com/paper/2404.09640/) (2024)
- [Uncertainty-Guided Appearance-Motion Association Network for Out-of-Distribution Action Detection](https://mlsanity.com/paper/2409.09953/) (2024)

## Abstract (arXiv)

In a real-world scenario, human actions are typically out of the distribution from training data, which requires a model to both recognize the known actions and reject the unknown. Different from image data, video actions are more challenging to be recognized in an open-set setting due to the uncertain temporal dynamics and static bias of human actions. In this paper, we propose a Deep Evidential Action Recognition (DEAR) method to recognize actions in an open testing set. Specifically, we formulate the action recognition problem from the evidential deep learning (EDL) perspective and propose a novel model calibration method to regularize the EDL training. Besides, to mitigate the static bias of video representation, we propose a plug-and-play module to debias the learned representation through contrastive learning. Experimental results show that our DEAR method achieves consistent performance gain on multiple mainstream action recognition models and benchmarks. Code and pre-trained models are available at smallurlhttps://www.rit.edu/actionlab/dear.