# Visual prompt engineering for video models

arXiv: https://arxiv.org/abs/2607.25537 · HTML: https://mlsanity.com/paper/2607.25537/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 44 references that are in the map.
- First posted on arXiv 28 Jul 2026.

## Builds on (most cited first, 30 of 32)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Wan: Open and Advanced Large-Scale Video Generative Models](https://mlsanity.com/paper/2503.20314/) (2025)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigm](https://mlsanity.com/paper/2102.07350/) (2021)
- [Unmasking Clever Hans Predictors and Assessing What Machines Really Learn](https://mlsanity.com/paper/1902.10178/) (2019)
- [Images Speak in Images: A Generalist Painter for In-Context Visual Learning](https://mlsanity.com/paper/2212.02499/) (2022)
- [Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP](https://mlsanity.com/paper/2212.14024/) (2022)
- [Exploring Visual Prompts for Adapting Large-Scale Models](https://mlsanity.com/paper/2203.17274/) (2022)
- [TextGrad: Automatic "Differentiation" via Text](https://mlsanity.com/paper/2406.07496/) (2024)
- [Visual Prompting via Image Inpainting](https://mlsanity.com/paper/2209.00647/) (2022)
- [Video models are zero-shot learners and reasoners](https://mlsanity.com/paper/2509.20328/) (2025)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [Cosmos 3: Omnimodal World Models for Physical AI](https://mlsanity.com/paper/2606.02800/) (2026)
- [Draw-and-Understand: Leveraging Visual Prompts to Enable MLLMs to Comprehend What You Want](https://mlsanity.com/paper/2403.20271/) (2024)
- [Video as the New Language for Real-World Decision Making](https://mlsanity.com/paper/2402.17139/) (2024)
- [Review of Large Vision Models and Visual Prompt Engineering](https://mlsanity.com/paper/2307.00855/) (2023)
- [Inducing Relational Knowledge from BERT](https://mlsanity.com/paper/1911.12753/) (2019)
- [Are Video Models Ready as Zero-Shot Reasoners? An Empirical Study with the MME-CoF Benchmark](https://mlsanity.com/paper/2510.26802/) (2025)
- [Thinking with Video: Video Generation as a Promising Multimodal Reasoning Paradigm](https://mlsanity.com/paper/2511.04570/) (2025)
- [VisualCloze: A Universal Image Generation Framework via Visual In-Context Learning](https://mlsanity.com/paper/2504.07960/) (2025)
- [A Very Big Video Reasoning Suite](https://mlsanity.com/paper/2602.20159/) (2026)
- [Demystifying Video Reasoning](https://mlsanity.com/paper/2603.16870/) (2026)
- [Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning](https://mlsanity.com/paper/2601.21037/) (2026)
- [Video Models Reason Early: Exploiting Plan Commitment for Maze Solving](https://mlsanity.com/paper/2603.30043/) (2026)
- [MentisOculi: Revealing the Limits of Reasoning with Mental Imagery](https://mlsanity.com/paper/2602.02465/) (2026)
- [VLMs are Good Teachers for Video Reasoning via Adaptive Test-Time Optimization](https://mlsanity.com/paper/2606.02564/) (2026)

## Built on by (most cited first, 1 of 1)

- [ProAR: Learning Prospective Reasoning with Autoregressive Video Models](https://mlsanity.com/paper/2610.03664/) (2026)

## Abstract (arXiv)

In the age of foundation models, a model is only as good as its prompt. For this reason, prompt engineering has become an essential technique for improving language model performance. Since video models are currently becoming foundation models for visual tasks (e.g., visual reasoning), we here ask whether they similarly benefit from visual prompt engineering: automatically modifying the task image to improve model performance. For example, for a visual physics reasoning task ("Where does the ball land, after passing a set of obstacles?"), an abstract sketch-like scene can be turned into a photorealistic version with a simple call to an image editing model. We find that visual prompt engineering, or VIPE for short, improves video reasoning performance across tasks. In fact, for video models, visual prompt engineering can be even more effective than classic text-based prompt engineering or test-time scaling. Ultimately, just as text-based prompt engineering systematically improves language model performance, visual prompt engineering can serve as a simple, compute-efficient approach to elicit better visual reasoning performance from video models. Example videos on our project page at https://visual-prompt-engineering.github.io/.