# CLIP meets GamePhysics: Towards bug identification in gameplay videos using zero-shot transfer learning

arXiv: https://arxiv.org/abs/2203.11096 · HTML: https://mlsanity.com/paper/2203.11096/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 61 references that are in the map.
- First posted on arXiv 21 Mar 2022.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [The Pushshift Reddit Dataset](https://mlsanity.com/paper/2001.08435/) (2020)
- [Unity: A General Platform for Intelligent Agents](https://mlsanity.com/paper/1809.02627/) (2018)
- [Strike (with) a Pose: Neural Networks Are Easily Fooled by Strange Poses of Familiar Objects](https://mlsanity.com/paper/1811.11553/) (2018)
- [Augmenting Automated Game Testing with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.15819/) (2021)
- [Deep Learning for Video Game Playing](https://mlsanity.com/paper/1708.07902/) (2017)
- [Improving Playtesting Coverage via Curiosity Driven Reinforcement Learning Agents](https://mlsanity.com/paper/2103.13798/) (2021)
- [Co-Creative Level Design via Machine Learning](https://mlsanity.com/paper/1809.09420/) (2018)
- [A Survey of Video Game Testing](https://mlsanity.com/paper/2103.06431/) (2021)
- [Contrastive Learning of Generalized Game Representations](https://mlsanity.com/paper/2106.10060/) (2021)
- [Entity Embedding as Game Representation](https://mlsanity.com/paper/2010.01685/) (2020)
- [Towards Creating a Deployable Grasp Type Probability Estimator for a Prosthetic Hand](https://mlsanity.com/paper/2101.05357/) (2021)
- [Predicting Game Engagement and Difficulty Using AI Players](https://mlsanity.com/paper/2107.12061/) (2021)
- [Computer Games Are Serious Business and so is their Quality: Particularities of Software Testing in Game Development from the Perspective of Practitioners](https://mlsanity.com/paper/1812.05164/) (2018)
- [NetCut: Real-Time DNN Inference Using Layer Removal](https://mlsanity.com/paper/2101.05363/) (2021)
- [Making CNNs for Video Parsing Accessible](https://mlsanity.com/paper/1906.11877/) (2019)
- [Dataset of Video Game Development Problems](https://mlsanity.com/paper/2001.00491/) (2020)
- [Explainability via Responsibility](https://mlsanity.com/paper/2010.01676/) (2020)
- [Under the Skin of Foundation NFT Auctions](https://mlsanity.com/paper/2109.12321/) (2021)
- [Identifying Similar Test Cases That Are Specified in Natural Language](https://mlsanity.com/paper/2110.07733/) (2021)

## Built on by (most cited first, 15 of 15)

- [GlitchBench: Can large multimodal models detect video game glitches?](https://mlsanity.com/paper/2312.05291/) (2023)
- [PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videos](https://mlsanity.com/paper/2412.01800/) (2024)
- [Large Language Models are Pretty Good Zero-Shot Video Game Bug Detectors](https://mlsanity.com/paper/2210.02506/) (2022)
- [Biases Propagate in Encoder-based Vision-Language Models: A Systematic Analysis From Intrinsic Measures to Zero-shot Retrieval Outcomes](https://mlsanity.com/paper/2506.06506/) (2025)
- [BehAVE: Behaviour Alignment of Video Game Encodings](https://mlsanity.com/paper/2402.01335/) (2024)
- [OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer](https://mlsanity.com/paper/2604.24762/) (2026)
- [Open-Ended Video Game Glitch Detection with Agentic Reasoning and Temporal Grounding](https://mlsanity.com/paper/2604.07818/) (2026)
- [RefGlitch-Bench: A Benchmark for Reference-based Gameplay Glitch Detection with Vision-Language Models](https://mlsanity.com/paper/2604.11082/) (2026)
- [InPhyRe Discovers: Large Multimodal Models Struggle in Inductive Physical Reasoning](https://mlsanity.com/paper/2509.12263/) (2025)
- [PhysEditWorld: A Large-Scale Dataset Toward Physics-Editable World Models](https://mlsanity.com/paper/2606.26694/) (2026)
- [Order from Chaos: Physical World Understanding from Glitchy Gameplay Videos](https://mlsanity.com/paper/2601.16471/) (2026)
- [How Far Can VLMs Go for Visual Bug Detection? Studying 19,738 Keyframes from 41 Hours of Gameplay Videos](https://mlsanity.com/paper/2603.22706/) (2026)
- [Runtime Failure Hunting for Physics Engine Based Software Systems: How Far Can We Go?](https://mlsanity.com/paper/2507.22099/) (2025)
- [EA-Swin: An Embedding-Agnostic Swin Transformer for AI-Generated Video Detection](https://mlsanity.com/paper/2602.17260/) (2026)
- [TempGlitch: Evaluating Vision-Language Models for Temporal Glitch Detection in Gameplay Videos](https://mlsanity.com/paper/2605.21443/) (2026)

## Abstract (arXiv)

Gameplay videos contain rich information about how players interact with the game and how the game responds. Sharing gameplay videos on social media platforms, such as Reddit, has become a common practice for many players. Often, players will share gameplay videos that showcase video game bugs. Such gameplay videos are software artifacts that can be utilized for game testing, as they provide insight for bug analysis. Although large repositories of gameplay videos exist, parsing and mining them in an effective and structured fashion has still remained a big challenge. In this paper, we propose a search method that accepts any English text query as input to retrieve relevant videos from large repositories of gameplay videos. Our approach does not rely on any external information (such as video metadata); it works solely based on the content of the video. By leveraging the zero-shot transfer capabilities of the Contrastive Language-Image Pre-Training (CLIP) model, our approach does not require any data labeling or training. To evaluate our approach, we present the textttGamePhysics dataset consisting of 26,954 videos from 1,873 games, that were collected from the GamePhysics section on the Reddit website. Our approach shows promising results in our extensive analysis of simple queries, compound queries, and bug queries, indicating that our approach is useful for object and event detection in gameplay videos. An example application of our approach is as a gameplay video search engine to aid in reproducing video game bugs. Please visit the following link for the code and the data: https://asgaardlab.github.io/CLIPxGamePhysics/