# Auto-captions on GIF: A Large-scale Video-sentence Dataset for Vision-language Pre-training

arXiv: https://arxiv.org/abs/2007.02375 · HTML: https://mlsanity.com/paper/2007.02375/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2020.

## Built on by (most cited first, 22 of 22)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [CLIP4Caption: CLIP for Video Caption](https://mlsanity.com/paper/2110.06615/) (2021)
- [Exploring Structure-aware Transformer over Interaction Proposals for Human-Object Interaction Detection](https://mlsanity.com/paper/2206.06291/) (2022)
- [Semantic-Conditional Diffusion Networks for Image Captioning](https://mlsanity.com/paper/2212.03099/) (2022)
- [Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrieval](https://mlsanity.com/paper/2201.09168/) (2022)
- [A Survey on Temporal Sentence Grounding in Videos](https://mlsanity.com/paper/2109.08039/) (2021)
- [Survey: Transformer based Video-Language Pre-training](https://mlsanity.com/paper/2109.09920/) (2021)
- [Understanding Chinese Video and Language via Contrastive Multimodal Pre-Training](https://mlsanity.com/paper/2104.09411/) (2021)
- [CoCo-BERT: Improving Video-Language Pre-training with Contrastive Cross-modal Matching and Denoising](https://mlsanity.com/paper/2112.07515/) (2021)
- [Scheduled Sampling in Vision-Language Pretraining with Decoupled Encoder-Decoder Network](https://mlsanity.com/paper/2101.11562/) (2021)
- [3DStyle-Diffusion: Pursuing Fine-grained Text-driven 3D Stylization with 2D Diffusion Models](https://mlsanity.com/paper/2311.05464/) (2023)
- [X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics](https://mlsanity.com/paper/2108.08217/) (2021)
- [Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training](https://mlsanity.com/paper/2201.04026/) (2022)
- [Bi-Calibration Networks for Weakly-Supervised Video Representation Learning](https://mlsanity.com/paper/2206.10491/) (2022)
- [View while Moving: Efficient Video Recognition in Long-untrimmed Videos](https://mlsanity.com/paper/2308.04834/) (2023)
- [Learning Sparsity for Effective and Efficient Music Performance Question Answering](https://mlsanity.com/paper/2506.01319/) (2025)
- [Silver-Bullet-3D at ManiSkill 2021: Learning-from-Demonstrations and Heuristic Rule-based Methods for Object Manipulation](https://mlsanity.com/paper/2206.06289/) (2022)
- [Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data](https://mlsanity.com/paper/2304.02080/) (2023)
- [Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation](https://mlsanity.com/paper/2112.14088/) (2021)
- [Extended Self-Critical Pipeline for Transforming Videos to Text (TRECVID-VTT Task 2021) -- Team: MMCUniAugsburg](https://mlsanity.com/paper/2112.14100/) (2021)
- [SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks](https://mlsanity.com/paper/2401.17773/) (2024)

## Abstract (arXiv)

In this work, we present Auto-captions on GIF, which is a new large-scale pre-training dataset for generic video understanding. All video-sentence pairs are created by automatically extracting and filtering video caption annotations from billions of web pages. Auto-captions on GIF dataset can be utilized to pre-train the generic feature representation or encoder-decoder structure for video captioning, and other downstream tasks (e.g., sentence localization in videos, video question answering, etc.) as well. We present a detailed analysis of Auto-captions on GIF dataset in comparison to existing video-sentence datasets. We also provide an evaluation of a Transformer-based encoder-decoder structure for vision-language pre-training, which is further adapted to video captioning downstream task and yields the compelling generalizability on MSR-VTT. The dataset is available at urlhttp://www.auto-video-captions.top/2020/dataset.