# ZBS: Zero-shot Background Subtraction via Instance-level Background Modeling and Foreground Selection

arXiv: https://arxiv.org/abs/2303.14679 · HTML: https://mlsanity.com/paper/2303.14679/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 51 references that are in the map.
- First posted on arXiv 26 Mar 2023.

## Builds on (most cited first, 18 of 18)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Simple Online and Realtime Tracking](https://mlsanity.com/paper/1602.00763/) (2016)
- [Detecting Twenty-thousand Classes using Image-level Supervision](https://mlsanity.com/paper/2201.02605/) (2022)
- [Zero-Shot Object Detection](https://mlsanity.com/paper/1804.04340/) (2018)
- [Background Subtraction in Real Applications: Challenges, Current Models and Future Directions](https://mlsanity.com/paper/1901.03577/) (2019)
- [Learning Multi-scale Features for Foreground Segmentation](https://mlsanity.com/paper/1808.01477/) (2018)
- [Deep Neural Network Concepts for Background Subtraction: A Systematic Review and Comparative Evaluation](https://mlsanity.com/paper/1811.05255/) (2018)
- [Learning to Measure Change: Fully Convolutional Siamese Metric Networks for Scene Change Detection](https://mlsanity.com/paper/1810.09111/) (2018)
- [BSUV-Net: A Fully-Convolutional Neural Network for Background Subtraction of Unseen Videos](https://mlsanity.com/paper/1907.11371/) (2019)
- [Foreground Segmentation Using a Triplet Convolutional Neural Network for Multiscale Feature Encoding](https://mlsanity.com/paper/1801.02225/) (2018)
- [BSUV-Net 2.0: Spatio-Temporal Data Augmentations for Video-Agnostic Supervised Background Subtraction](https://mlsanity.com/paper/2101.09585/) (2021)
- [Universal Background Subtraction based on Arithmetic Distribution Neural Network](https://mlsanity.com/paper/2104.08390/) (2021)
- [WisenetMD: Motion Detection Using Dynamic Background Region Analysis](https://mlsanity.com/paper/1805.09277/) (2018)
- [Autoencoder-based background reconstruction and foreground segmentation with background noise estimation](https://mlsanity.com/paper/2112.08001/) (2021)
- [Background Subtraction with Real-time Semantic Segmentation](https://mlsanity.com/paper/1811.10020/) (2018)
- [Joint Background Reconstruction and Foreground Segmentation via A Two-stage Convolutional Neural Network](https://mlsanity.com/paper/1707.07584/) (2017)
- [Dynamic Background Subtraction by Generative Neural Networks](https://mlsanity.com/paper/2202.05336/) (2022)

## Built on by (most cited first, 9 of 9)

- [Instance-Level Moving Object Segmentation from a Single Image with Events](https://mlsanity.com/paper/2502.12975/) (2025)
- [LangGas: Introducing Language in Selective Zero-Shot Background Subtraction for Semi-Transparent Gas Leak Detection with a New Dataset](https://mlsanity.com/paper/2503.02910/) (2025)
- [FlexCache: Flexible Approximate Cache System for Video Diffusion](https://mlsanity.com/paper/2501.04012/) (2024)
- [Learning Temporal Distribution and Spatial Correlation Towards Universal Moving Object Segmentation](https://mlsanity.com/paper/2304.09949/) (2023)
- [Learning Spatial-Temporal Regularized Tensor Sparse RPCA for Background Subtraction](https://mlsanity.com/paper/2309.15576/) (2023)
- [Review of Zero-Shot and Few-Shot AI Algorithms in The Medical Domain](https://mlsanity.com/paper/2406.16143/) (2024)
- [ZS-VCOS: Zero-Shot Video Camouflaged Object Segmentation By Optical Flow and Open Vocabulary Object Detection](https://mlsanity.com/paper/2505.01431/) (2025)
- [SAEN-BGS: Energy-Efficient Spiking AutoEncoder Network for Background Subtraction](https://mlsanity.com/paper/2505.07336/) (2025)
- [MUSTAN: Multi-scale Temporal Context as Attention for Robust Video Foreground Segmentation](https://mlsanity.com/paper/2402.00918/) (2024)

## Abstract (arXiv)

Background subtraction (BGS) aims to extract all moving objects in the video frames to obtain binary foreground segmentation masks. Deep learning has been widely used in this field. Compared with supervised-based BGS methods, unsupervised methods have better generalization. However, previous unsupervised deep learning BGS algorithms perform poorly in sophisticated scenarios such as shadows or night lights, and they cannot detect objects outside the pre-defined categories. In this work, we propose an unsupervised BGS algorithm based on zero-shot object detection called Zero-shot Background Subtraction (ZBS). The proposed method fully utilizes the advantages of zero-shot object detection to build the open-vocabulary instance-level background model. Based on it, the foreground can be effectively extracted by comparing the detection results of new frames with the background model. ZBS performs well for sophisticated scenarios, and it has rich and extensible categories. Furthermore, our method can easily generalize to other tasks, such as abandoned object detection in unseen environments. We experimentally show that ZBS surpasses state-of-the-art unsupervised BGS methods by 4.70% F-Measure on the CDnet 2014 dataset. The code is released at https://github.com/CASIA-IVA-Lab/ZBS.