# What Can We Learn from Harry Potter? An Exploratory Study of Visual Representation Learning from Atypical Videos

arXiv: https://arxiv.org/abs/2508.21770 · HTML: https://mlsanity.com/paper/2508.21770/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 45 of its 60 references that are in the map.
- First posted on arXiv 29 Aug 2025.

## Builds on (most cited first, 30 of 45)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [Real-world Anomaly Detection in Surveillance Videos](https://mlsanity.com/paper/1801.04264/) (2018)
- [Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet?](https://mlsanity.com/paper/1711.09577/) (2017)
- [Towards Open Set Deep Networks](https://mlsanity.com/paper/1511.06233/) (2015)
- [Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data](https://mlsanity.com/paper/2002.11297/) (2020)
- [Learning from Synthetic Humans](https://mlsanity.com/paper/1701.01370/) (2017)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [Expanding Language-Image Pretrained Models for General Video Recognition](https://mlsanity.com/paper/2208.02816/) (2022)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [OpenOOD: Benchmarking Generalized Out-of-Distribution Detection](https://mlsanity.com/paper/2210.07242/) (2022)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Learning to Discover Novel Visual Categories via Deep Transfer Clustering](https://mlsanity.com/paper/1908.09884/) (2019)
- [Moments in Time Dataset: one million videos for event understanding](https://mlsanity.com/paper/1801.03150/) (2018)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [Fishr: Invariant Gradient Variances for Out-of-Distribution Generalization](https://mlsanity.com/paper/2109.02934/) (2021)
- [Human Action Recognition from Various Data Modalities: A Review](https://mlsanity.com/paper/2012.11866/) (2020)
- [Automatically Discovering and Learning New Visual Categories with Ranking Statistics](https://mlsanity.com/paper/2002.05714/) (2020)
- [Self-supervised Video Representation Learning by Pace Prediction](https://mlsanity.com/paper/2008.05861/) (2020)
- [OpenGAN: Open-Set Recognition via Open Data Generation](https://mlsanity.com/paper/2104.02939/) (2021)
- [Learning Placeholders for Open-Set Recognition](https://mlsanity.com/paper/2103.15086/) (2021)
- [AutoNovel: Automatically Discovering and Learning Novel Visual Categories](https://mlsanity.com/paper/2106.15252/) (2021)
- [POEM: Out-of-Distribution Detection with Posterior Sampling](https://mlsanity.com/paper/2206.13687/) (2022)
- [OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in An Open World](https://mlsanity.com/paper/2004.05551/) (2020)

## Abstract (arXiv)

Humans usually show exceptional generalisation and discovery ability in the open world, when being shown uncommon new concepts. Whereas most existing studies in the literature focus on common typical data from closed sets, open-world novel discovery is under-explored in videos. In this paper, we are interested in asking: What if atypical unusual videos are exposed in the learning process? To this end, we collect a new video dataset consisting of various types of unusual atypical data (e.g., sci-fi, animation, etc.). To study how such atypical data may benefit open-world learning, we feed them into the model training process for representation learning. Focusing on three key tasks in open-world learning: out-of-distribution (OOD) detection, novel category discovery (NCD), and zero-shot action recognition (ZSAR), we found that even straightforward learning approaches with atypical data consistently improve performance across various settings. Furthermore, we found that increasing the categorical diversity of the atypical samples further boosts OOD detection performance. Additionally, in the NCD task, using a smaller yet more semantically diverse set of atypical samples leads to better performance compared to using a larger but more typical dataset. In the ZSAR setting, the semantic diversity of atypical videos helps the model generalise better to unseen action classes. These observations in our extensive experimental evaluations reveal the benefits of atypical videos for visual representation learning in the open world, together with the newly proposed dataset, encouraging further studies in this direction. The project page is at: https://julysun98.github.io/atypical_dataset.