# VideoCLIP-XL: Advancing Long Description Understanding for Video CLIP Models

arXiv: https://arxiv.org/abs/2410.00741 · HTML: https://mlsanity.com/paper/2410.00741/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 65 references that are in the map.
- First posted on arXiv 1 Oct 2024.

## Builds on (most cited first, 30 of 33)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [LAION-5B: An open large-scale dataset for training next generation image-text models](https://mlsanity.com/paper/2210.08402/) (2022)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts](https://mlsanity.com/paper/2102.08981/) (2021)
- [Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrieval](https://mlsanity.com/paper/2104.00650/) (2021)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clips](https://mlsanity.com/paper/1906.03327/) (2019)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [GroupViT: Semantic Segmentation Emerges from Text Supervision](https://mlsanity.com/paper/2202.11094/) (2022)
- [InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2307.06942/) (2023)
- [VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research](https://mlsanity.com/paper/1904.03493/) (2019)
- [Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teachers](https://mlsanity.com/paper/2402.19479/) (2024)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts](https://mlsanity.com/paper/2111.08276/) (2021)
- [Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptions](https://mlsanity.com/paper/2111.10337/) (2021)
- [Unmasked Teacher: Towards Training-Efficient Video Foundation Models](https://mlsanity.com/paper/2303.16058/) (2023)
- [A Dataset for Movie Description](https://mlsanity.com/paper/1501.02530/) (2015)
- [CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoders](https://mlsanity.com/paper/2106.14843/) (2021)
- [Alpha-CLIP: A CLIP Model Focusing on Wherever You Want](https://mlsanity.com/paper/2312.03818/) (2023)
- [Region-Aware Pretraining for Open-Vocabulary Object Detection with Vision Transformers](https://mlsanity.com/paper/2305.07011/) (2023)
- [Tag2Text: Guiding Vision-Language Model via Image Tagging](https://mlsanity.com/paper/2303.05657/) (2023)

## Built on by (most cited first, 30 of 41)

- [LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agents](https://mlsanity.com/paper/2503.10200/) (2025)
- [UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions](https://mlsanity.com/paper/2506.13691/) (2025)
- [IVEBench: Modern Benchmark Suite for Instruction-Guided Video Editing Assessment](https://mlsanity.com/paper/2510.11647/) (2025)
- [T2AV-Compass: Towards Unified Evaluation for Text-to-Audio-Video Generation](https://mlsanity.com/paper/2512.21094/) (2025)
- [Embed-RL: Reinforcement Learning for Reasoning-Driven Multimodal Embeddings](https://mlsanity.com/paper/2602.13823/) (2026)
- [PanoWan: Lifting Diffusion Video Generation Models to 360deg with Latitude/Longitude-aware Mechanisms](https://mlsanity.com/paper/2505.22016/) (2025)
- [Dexterous World Models](https://mlsanity.com/paper/2512.17907/) (2025)
- [Audio-Sync Video Generation with Multi-Stream Temporal Control](https://mlsanity.com/paper/2506.08003/) (2025)
- [LoVR: A Benchmark for Long Video Retrieval in Multimodal Contexts](https://mlsanity.com/paper/2505.13928/) (2025)
- [Integrating Reinforcement Learning with Visual Generative Models: Foundations and Advances](https://mlsanity.com/paper/2508.10316/) (2025)
- [VTimeCoT: Thinking by Drawing for Video Temporal Grounding and Reasoning](https://mlsanity.com/paper/2510.14672/) (2025)
- [SkyReels-V4: Multi-modal Video-Audio Generation, Inpainting and Editing model](https://mlsanity.com/paper/2602.21818/) (2026)
- [Towards Universal Video Retrieval: Generalizing Video Embedding via Synthesized Multimodal Pyramid Curriculum](https://mlsanity.com/paper/2510.27571/) (2025)
- [AVI-Edit: Audio-sync Video Instance Editing with Granularity-Aware Mask Refiner](https://mlsanity.com/paper/2512.10571/) (2025)
- [TeleBoost: A Systematic Alignment Framework for High-Fidelity, Controllable, and Robust Video Generation](https://mlsanity.com/paper/2602.07595/) (2026)
- [Can video generation replace cinematographers? Research on the cinematic language of generated video](https://mlsanity.com/paper/2412.12223/) (2024)
- [LumosX: Relate Any Identities with Their Attributes for Personalized Video Generation](https://mlsanity.com/paper/2603.20192/) (2026)
- [ContentV: Efficient Training of Video Generation Models with Limited Compute](https://mlsanity.com/paper/2506.05343/) (2025)
- [Learning What to Trust: Bayesian Prior-Guided Optimization for Visual Generation](https://mlsanity.com/paper/2511.18919/) (2025)
- [VKnowU: Evaluating Visual Knowledge Understanding in Multimodal LLMs](https://mlsanity.com/paper/2511.20272/) (2025)
- [InstructAV2AV: Instruction-Guided Audio-Video Joint Editing](https://mlsanity.com/paper/2605.18467/) (2026)
- [InteractiveAvatar: Real-Time Streaming Video Generation for Consistent and Intent-Aware Avatars](https://mlsanity.com/paper/2606.22905/) (2026)
- [Video-CoE: Reinforcing Video Event Prediction via Chain of Events](https://mlsanity.com/paper/2603.14935/) (2026)
- [VIRES: Video Instance Repainting via Sketch and Text Guided Generation](https://mlsanity.com/paper/2411.16199/) (2024)
- [TRoVe: Discovering Error-Inducing Static Feature Biases in Temporal Vision-Language Models](https://mlsanity.com/paper/2512.01048/) (2025)
- [Seeing Realism from Simulation: Efficient Video Transfer for Vision-Language-Action Data Augmentation](https://mlsanity.com/paper/2605.02757/) (2026)
- [Sample-efficient Integration of New Modalities into Large Language Models](https://mlsanity.com/paper/2509.04606/) (2025)
- [NeoWorld: Neural Simulation of Explorable Virtual Worlds via Progressive 3D Unfolding](https://mlsanity.com/paper/2509.24441/) (2025)
- [XEmoGPT: An Explainable Multimodal Emotion Recognition Framework with Cue-Level Perception and Reasoning](https://mlsanity.com/paper/2602.05496/) (2026)
- [Binding Multiple Modalities via Multimodal Wasserstein Barycenter](https://mlsanity.com/paper/2609.33800/) (2026)

## Abstract (arXiv)

Contrastive Language-Image Pre-training (CLIP) has been widely studied and applied in numerous applications. However, the emphasis on brief summary texts during pre-training prevents CLIP from understanding long descriptions. This issue is particularly acute regarding videos given that videos often contain abundant detailed contents. In this paper, we propose the VideoCLIP-XL (eXtra Length) model, which aims to unleash the long-description understanding capability of video CLIP models. Firstly, we establish an automatic data collection system and gather a large-scale VILD pre-training dataset with VIdeo and Long-Description pairs. Then, we propose Text-similarity-guided Primary Component Matching (TPCM) to better learn the distribution of feature space while expanding the long description capability. We also introduce two new tasks namely Detail-aware Description Ranking (DDR) and Hallucination-aware Description Ranking (HDR) for further understanding improvement. Finally, we construct a Long Video Description Ranking (LVDR) benchmark for evaluating the long-description capability more comprehensively. Extensive experimental results on widely-used text-video retrieval benchmarks with both short and long descriptions and our LVDR benchmark can fully demonstrate the effectiveness of our method.