# VideoCon: Robust Video-Language Alignment via Contrast Captions

arXiv: https://arxiv.org/abs/2311.10111 · HTML: https://mlsanity.com/paper/2311.10111/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 56 of its 74 references that are in the map.
- First posted on arXiv 15 Nov 2023.

## Builds on (most cited first, 30 of 56)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Zero-Shot Text-to-Image Generation](https://mlsanity.com/paper/2102.12092/) (2021)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2304.08818/) (2023)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [The "something something" video database for learning and evaluating visual common sense](https://mlsanity.com/paper/1706.04261/) (2017)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [VideoChat: Chat-Centric Video Understanding](https://mlsanity.com/paper/2305.06355/) (2023)
- [NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actions](https://mlsanity.com/paper/2105.08276/) (2021)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [ModelScope Text-to-Video Technical Report](https://mlsanity.com/paper/2308.06571/) (2023)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [DataComp: In search of the next generation of multimodal datasets](https://mlsanity.com/paper/2304.14108/) (2023)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [When and why vision-language models behave like bags-of-words, and what to do about it?](https://mlsanity.com/paper/2210.01936/) (2022)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023)

## Built on by (most cited first, 30 of 30)

- [VideoPhy: Evaluating Physical Commonsense for Video Generation](https://mlsanity.com/paper/2406.03520/) (2024)
- [DreamGen: Unlocking Generalization in Robot Learning through Video World Models](https://mlsanity.com/paper/2505.12705/) (2025)
- [VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation](https://mlsanity.com/paper/2503.06800/) (2025)
- [VideoPrism: A Foundational Visual Encoder for Video Understanding](https://mlsanity.com/paper/2402.13217/) (2024)
- [Exploring the Evolution of Physics Cognition in Video Generation: A Survey](https://mlsanity.com/paper/2503.21765/) (2025)
- [Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainability](https://mlsanity.com/paper/2501.01346/) (2025)
- [TALC: Time-Aligned Captions for Multi-Scene Text-to-Video Generation](https://mlsanity.com/paper/2405.04682/) (2024)
- [VidHal: Benchmarking Temporal Hallucinations in Vision LLMs](https://mlsanity.com/paper/2411.16771/) (2024)
- [Do Egocentric Video-Language Models Truly Understand Hand-Object Interactions?](https://mlsanity.com/paper/2405.17719/) (2024)
- [VELOCITI: Benchmarking Video-Language Compositional Reasoning with Strict Entailment](https://mlsanity.com/paper/2406.10889/) (2024)
- [Direct Motion Models for Assessing Generated Videos](https://mlsanity.com/paper/2505.00209/) (2025)
- [UVE: Are MLLMs Unified Evaluators for AI-Generated Videos?](https://mlsanity.com/paper/2503.09949/) (2025)
- [NL-Eye: Abductive NLI for Images](https://mlsanity.com/paper/2410.02613/) (2024)
- [Know-Show: Benchmarking Video-Language Models on Spatio-Temporal Grounded Reasoning](https://mlsanity.com/paper/2512.05513/) (2025)
- [Expertized Caption Auto-Enhancement for Video-Text Retrieval](https://mlsanity.com/paper/2502.02885/) (2025)
- [Can LLMs Deceive CLIP? Benchmarking Adversarial Compositionality of Pre-trained Multimodal Representation via Text Updates](https://mlsanity.com/paper/2505.22943/) (2025)
- [MESH -- Understanding Videos Like Human: Measuring Hallucinations in Large Video Models](https://mlsanity.com/paper/2509.08538/) (2025)
- [Can Text-to-Video Generation help Video-Language Alignment?](https://mlsanity.com/paper/2503.18507/) (2025)
- [MINERVA-Cultural: A Benchmark for Cultural and Multilingual Long Video Reasoning](https://mlsanity.com/paper/2601.10649/) (2026)
- [Learning to Decode Against Compositional Hallucination in Video Multimodal Large Language Models](https://mlsanity.com/paper/2602.00559/) (2026)
- [Can Hallucination Correction Improve Video-Language Alignment?](https://mlsanity.com/paper/2502.15079/) (2025)
- [REVEAL: Relation-based Video Representation Learning for Video-Question-Answering](https://mlsanity.com/paper/2504.05463/) (2025)
- [Harnessing Synthetic Preference Data for Enhancing Temporal Understanding of Video-LLMs](https://mlsanity.com/paper/2510.03955/) (2025)
- [Exploring the AI Obedience: Why is Generating a Pure Color Image Harder than CyberPunk?](https://mlsanity.com/paper/2603.00166/) (2026)
- [InstAP: Instance-Aware Vision-Language Pre-Train for Spatial-Temporal Understanding](https://mlsanity.com/paper/2604.08337/) (2026)
- [MultiToP: Learning to Patch Visual Tokens to Mitigate Hallucinations in Video Large Multimodal Models](https://mlsanity.com/paper/2606.11792/) (2026)
- [Does This Moment Justify the Recommendation? Counterfactual Behavior-Grounded Evidence Retrieval for Personalized Video Recommendation](https://mlsanity.com/paper/2609.00996/) (2026)
- [Not Another Text Benchmark: Putting the "Visual" Back in Visual Question Answering for Large Video Models](https://mlsanity.com/paper/2609.17112/) (2026)
- [Do World Models Make Better Robots? A Survey of Evaluation Benchmarks for Predictive Embodied Intelligence](https://mlsanity.com/paper/2609.29669/) (2026)
- [CineRecon: Benchmarking Frontier Models for Agentic Cinematic Reconstruction](https://mlsanity.com/paper/2610.16577/) (2026)

## Abstract (arXiv)

Despite being (pre)trained on a massive amount of data, state-of-the-art video-language alignment models are not robust to semantically-plausible contrastive changes in the video captions. Our work addresses this by identifying a broad spectrum of contrast misalignments, such as replacing entities, actions, and flipping event order, which alignment models should be robust against. To this end, we introduce the VideoCon, a video-language alignment dataset constructed by a large language model that generates plausible contrast video captions and explanations for differences between original and contrast video captions. Then, a generative video-language model is finetuned with VideoCon to assess video-language entailment and generate explanations. Our VideoCon-based alignment model significantly outperforms current models. It exhibits a 12-point increase in AUC for the video-language alignment task on human-generated contrast captions. Finally, our model sets new state of the art zero-shot performance in temporally-extensive video-language tasks such as text-to-video retrieval (SSv2-Temporal) and video question answering (ATP-Hard). Moreover, our model shows superior performance on novel videos and human-crafted captions and explanations. Our code and data are available at https://github.com/Hritikbansal/videocon.