# Poisoning Prompt-Guided Sampling in Video Large Language Models

arXiv: https://arxiv.org/abs/2509.20851 · HTML: https://mlsanity.com/paper/2509.20851/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 65 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 30 of 55)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling](https://mlsanity.com/paper/2412.05271/) (2024)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Video-LLaVA: Learning United Visual Representation by Alignment Before Projection](https://mlsanity.com/paper/2311.10122/) (2023)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [Universal adversarial perturbations](https://mlsanity.com/paper/1610.08401/) (2016)
- [VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMs](https://mlsanity.com/paper/2406.07476/) (2024)
- [Fast is better than free: Revisiting adversarial training](https://mlsanity.com/paper/2001.03994/) (2020)
- [LLaVA-Video: Video Instruction Tuning With Synthetic Data](https://mlsanity.com/paper/2410.02713/) (2024)
- [LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models](https://mlsanity.com/paper/2311.17043/) (2023)
- [VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding](https://mlsanity.com/paper/2501.13106/) (2025)
- [ShareGPT4Video: Improving Video Understanding and Generation with Better Captions](https://mlsanity.com/paper/2406.04325/) (2024)
- [Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding](https://mlsanity.com/paper/2311.08046/) (2023)
- [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023)
- [Certified Defenses for Data Poisoning Attacks](https://mlsanity.com/paper/1706.03691/) (2017)
- [Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioning](https://mlsanity.com/paper/2302.14115/) (2023)
- [VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on Long Videos](https://mlsanity.com/paper/2405.19209/) (2024)
- [Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision](https://mlsanity.com/paper/2007.04687/) (2020)
- [VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding](https://mlsanity.com/paper/2403.11481/) (2024)
- [NVILA: Efficient Frontier Visual Language Models](https://mlsanity.com/paper/2412.04468/) (2024)
- [Video Understanding with Large Language Models: A Survey](https://mlsanity.com/paper/2312.17432/) (2023)
- [Learning Video Representations from Large Language Models](https://mlsanity.com/paper/2212.04501/) (2022)
- [Self-Chained Image-Language Model for Video Localization and Question Answering](https://mlsanity.com/paper/2305.06988/) (2023)
- [A Simple LLM Framework for Long-Range Video Question-Answering](https://mlsanity.com/paper/2312.17235/) (2023)
- [Streaming Long Video Understanding with Large Language Models](https://mlsanity.com/paper/2405.16009/) (2024)

## Built on by (most cited first, 4 of 4)

- [VideoSTF: Stress-Testing Output Repetition in Video Large Language Models](https://mlsanity.com/paper/2602.10639/) (2026)
- [FeatureFool: Zero-Query Fooling of Video Models via Feature Map](https://mlsanity.com/paper/2510.18362/) (2025)
- [Membership Inference Attacks Against Video Large Language Models](https://mlsanity.com/paper/2604.27002/) (2026)
- [Do Input-Level Defenses Transfer to Observation-Level Attacks on VideoLLMs?](https://mlsanity.com/paper/2609.08331/) (2026)

## Abstract (arXiv)

Video Large Language Models (VideoLLMs) are increasingly deployed as automated moderators on user-generated video platforms, where a few unwatched seconds of harmful footage are enough to suppress a safety alert. Because encoding every frame is prohibitive, modern VideoLLMs rely on prompt-guided sampling (PGS), which scores frames against the user prompt and forwards only the top-ranked ones to the visual encoder. Uniform and semantic samplers are known to be defeated by simple frame replacement, whereas PGS, the most prompt-aware family, has escaped scrutiny, and its prompt awareness in fact repairs the omission failures that defeat the other two. We show that this repair is superficial, since PoisonVID, a transfer attack, poisons the sampler's ranking so that harmful clips are never surfaced, without access to target weights, gradients, or sampling internals. It optimizes one video-level perturbation under a relevance-suppression loss defined over a depiction set of paraphrased harmful descriptions written by a shadow VideoLLM and a general-purpose language model, which drives perturbed harmful frames out of the prompt-conditioned subspace that PGS reads. Samplers that never consult that score keep the frames they always kept, which locates the failure at selection rather than at the encoder. Across three PGS methods, six VideoLLMs, and six harmful categories, PoisonVID attains 84% to 97% average attack success over the 18 sampler and model pairs and survives seven defenses. Re-encoding at lower resolution on ingest gives back part of what was evicted and costs the attack 48 points, which bounds the threat without closing it. PGS therefore buys accuracy with a structural safety debt, and sampler design will now have to repay that.