# SOAR: Scene-debiasing Open-set Action Recognition

arXiv: https://arxiv.org/abs/2309.01265 · HTML: https://mlsanity.com/paper/2309.01265/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 61 of its 94 references that are in the map.
- First posted on arXiv 3 Sep 2023.

## Builds on (most cited first, 30 of 61)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Learning Deep Features for Discriminative Localization](https://mlsanity.com/paper/1512.04150/) (2015)
- [Non-local Neural Networks](https://mlsanity.com/paper/1711.07971/) (2017)
- [Unsupervised Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1409.7495/) (2014)
- [SlowFast Networks for Video Recognition](https://mlsanity.com/paper/1812.03982/) (2018)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [TSM: Temporal Shift Module for Efficient Video Understanding](https://mlsanity.com/paper/1811.08383/) (2018)
- [Evidential Deep Learning to Quantify Classification Uncertainty](https://mlsanity.com/paper/1806.01768/) (2018)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Deep Evidential Regression](https://mlsanity.com/paper/1910.02600/) (2019)
- [Recent Advances in Open Set Recognition: A Survey](https://mlsanity.com/paper/1811.08581/) (2018)
- [Temporal Segment Networks for Action Recognition in Videos](https://mlsanity.com/paper/1705.02953/) (2017)
- [Learning Not to Learn: Training Deep Neural Networks with Biased Data](https://mlsanity.com/paper/1812.10352/) (2018)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [Learning De-biased Representations with Biased Representations](https://mlsanity.com/paper/1910.02806/) (2019)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Learning Gender-Neutral Word Embeddings](https://mlsanity.com/paper/1809.01496/) (2018)
- [Temporal Pyramid Network for Action Recognition](https://mlsanity.com/paper/2004.03548/) (2020)
- [Generative OpenMax for Multi-Class Open Set Classification](https://mlsanity.com/paper/1707.07418/) (2017)
- [Adversarial Reciprocal Points Learning for Open Set Recognition](https://mlsanity.com/paper/2103.00953/) (2021)

## Built on by (most cited first, 12 of 12)

- [XOV-Action: Towards Generalizable Open-Vocabulary Action Recognition](https://mlsanity.com/paper/2403.01560/) (2024)
- [Fairness and Bias Mitigation in Computer Vision: A Survey](https://mlsanity.com/paper/2408.02464/) (2024)
- [Open-Set Biometrics: Beyond Good Closed-Set Models](https://mlsanity.com/paper/2407.16133/) (2024)
- [ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognition](https://mlsanity.com/paper/2502.00156/) (2025)
- [SMART: Scene-motion-aware human action recognition framework for mental disorder group](https://mlsanity.com/paper/2406.04649/) (2024)
- [Divide and Fuse: Body Part Mesh Recovery from Partially Visible Human Images](https://mlsanity.com/paper/2407.09694/) (2024)
- [ContextHOI: Spatial Context Learning for Human-Object Interaction Detection](https://mlsanity.com/paper/2412.09050/) (2024)
- [Unbiasing through Textual Descriptions: Mitigating Representation Bias in Video Benchmarks](https://mlsanity.com/paper/2503.18637/) (2025)
- [Multi-level and Multi-modal Action Anticipation](https://mlsanity.com/paper/2506.02382/) (2025)
- [CASR: Refining Action Segmentation via Marginalizing Frame-levle Causal Relationships](https://mlsanity.com/paper/2311.12401/) (2023)
- [Exploring Ordinal Bias in Action Recognition for Instructional Videos](https://mlsanity.com/paper/2504.06580/) (2025)
- [Learning to Deny: Action Denial in Multimodal Large Language Models](https://mlsanity.com/paper/2606.31187/) (2026)

## Abstract (arXiv)

Deep learning models have a risk of utilizing spurious clues to make predictions, such as recognizing actions based on the background scene. This issue can severely degrade the open-set action recognition performance when the testing samples have different scene distributions from the training samples. To mitigate this problem, we propose a novel method, called Scene-debiasing Open-set Action Recognition (SOAR), which features an adversarial scene reconstruction module and an adaptive adversarial scene classification module. The former prevents the decoder from reconstructing the video background given video features, and thus helps reduce the background information in feature learning. The latter aims to confuse scene type classification given video features, with a specific emphasis on the action foreground, and helps to learn scene-invariant information. In addition, we design an experiment to quantify the scene bias. The results indicate that the current open-set action recognizers are biased toward the scene, and our proposed SOAR method better mitigates such bias. Furthermore, our extensive experiments demonstrate that our method outperforms state-of-the-art methods, and the ablation studies confirm the effectiveness of our proposed modules.