# AutoLabel: CLIP-based framework for Open-set Video Domain Adaptation

arXiv: https://arxiv.org/abs/2304.01110 · HTML: https://mlsanity.com/paper/2304.01110/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 59 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Temporal Segment Networks: Towards Good Practices for Deep Action Recognition](https://mlsanity.com/paper/1608.00859/) (2016)
- [Scaling Egocentric Vision: The EPIC-KITCHENS Dataset](https://mlsanity.com/paper/1804.02748/) (2018)
- [Maximum Classifier Discrepancy for Unsupervised Domain Adaptation](https://mlsanity.com/paper/1712.02560/) (2017)
- [Deep Transfer Learning with Joint Adaptation Networks](https://mlsanity.com/paper/1605.06636/) (2016)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Long-term Recurrent Convolutional Networks for Visual Recognition and Description](https://mlsanity.com/paper/1411.4389/) (2014)
- [Convolutional Two-Stream Network Fusion for Video Action Recognition](https://mlsanity.com/paper/1604.06573/) (2016)
- [ActionCLIP: A New Paradigm for Video Action Recognition](https://mlsanity.com/paper/2109.08472/) (2021)
- [Video Action Transformer Network](https://mlsanity.com/paper/1812.02707/) (2018)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Human Action Recognition and Prediction: A Survey](https://mlsanity.com/paper/1806.11230/) (2018)
- [Towards Open World Recognition](https://mlsanity.com/paper/1412.5687/) (2014)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition](https://mlsanity.com/paper/1708.07632/) (2017)
- [Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP](https://mlsanity.com/paper/2109.02748/) (2021)
- [Open Set Domain Adaptation by Backpropagation](https://mlsanity.com/paper/1804.10427/) (2018)
- [Multi-Modal Domain Adaptation for Fine-Grained Action Recognition](https://mlsanity.com/paper/2001.09691/) (2020)
- [Temporal Attentive Alignment for Large-Scale Video Domain Adaptation](https://mlsanity.com/paper/1907.12743/) (2019)
- [OVANet: One-vs-All Network for Universal Domain Adaptation](https://mlsanity.com/paper/2104.03344/) (2021)
- [A Comprehensive Study of Deep Video Action Recognition](https://mlsanity.com/paper/2012.06567/) (2020)
- [Space-time Mixing Attention for Video Transformer](https://mlsanity.com/paper/2106.05968/) (2021)

## Built on by (most cited first, 18 of 18)

- [Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Survey](https://mlsanity.com/paper/2407.21794/) (2024)
- [EmoCLIP: A Vision-Language Method for Zero-Shot Video Facial Expression Recognition](https://mlsanity.com/paper/2310.16640/) (2023)
- [Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models](https://mlsanity.com/paper/2501.18592/) (2025)
- [CLIP-Powered Domain Generalization and Domain Adaptation: A Comprehensive Survey](https://mlsanity.com/paper/2504.14280/) (2025)
- [MissionGNN: Hierarchical Multimodal GNN-based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation](https://mlsanity.com/paper/2406.18815/) (2024)
- [Open-Set Recognition in the Age of Vision-Language Models](https://mlsanity.com/paper/2403.16528/) (2024)
- [LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition](https://mlsanity.com/paper/2309.12780/) (2023)
- [Recall and Refine: A Simple but Effective Source-free Open-set Domain Adaptation Framework](https://mlsanity.com/paper/2411.12558/) (2024)
- [Language-Guided Instance-Aware Domain-Adaptive Panoptic Segmentation](https://mlsanity.com/paper/2404.03799/) (2024)
- [Target Semantics Clustering via Text Representations for Robust Universal Domain Adaptation](https://mlsanity.com/paper/2506.03521/) (2025)
- [Preserving Clusters in Prompt Learning for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2506.11493/) (2025)
- [Open-Set Video-based Facial Expression Recognition with Human Expression-sensitive Prompting](https://mlsanity.com/paper/2404.17100/) (2024)
- [ICON: Improving Inter-Report Consistency in Radiology Report Generation via Lesion-aware Mixup Augmentation](https://mlsanity.com/paper/2402.12844/) (2024)
- [Trustworthy Representation Learning Across Domains](https://mlsanity.com/paper/2308.12315/) (2023)
- [Continuous GNN-based Anomaly Detection on Edge using Efficient Adaptive Knowledge Graph Learning](https://mlsanity.com/paper/2411.09072/) (2024)
- [Training-Free Label Space Alignment for Universal Domain Adaptation](https://mlsanity.com/paper/2509.17452/) (2025)
- [SemAlign: Language Guided Semi-supervised Domain Generalization](https://mlsanity.com/paper/2601.11724/) (2026)
- [Return of Frustratingly Easy Unsupervised Video Domain Adaptation](https://mlsanity.com/paper/2605.19510/) (2026)

## Abstract (arXiv)

Open-set Unsupervised Video Domain Adaptation (OUVDA) deals with the task of adapting an action recognition model from a labelled source domain to an unlabelled target domain that contains "target-private" categories, which are present in the target but absent in the source. In this work we deviate from the prior work of training a specialized open-set classifier or weighted adversarial learning by proposing to use pre-trained Language and Vision Models (CLIP). The CLIP is well suited for OUVDA due to its rich representation and the zero-shot recognition capabilities. However, rejecting target-private instances with the CLIP's zero-shot protocol requires oracle knowledge about the target-private label names. To circumvent the impossibility of the knowledge of label names, we propose AutoLabel that automatically discovers and generates object-centric compositional candidate target-private class names. Despite its simplicity, we show that CLIP when equipped with AutoLabel can satisfactorily reject the target-private instances, thereby facilitating better alignment between the shared classes of the two domains. The code is available.