mlsanityData to 7 Oct 2026

TGIF: A New Dataset and Benchmark on Animated GIF Description

arXiv 1604.02748 ↗First posted 10 Apr 2016cs.CVcited by 124

Not on the map. Why?

Cited by 29 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

With the recent popularity of animated GIFs on social media, there is need for ways to index them with rich metadata. To advance research on animated GIF understanding, we collected a new dataset, Tumblr GIF (TGIF), with 100K animated GIFs from Tumblr and 120K natural language descriptions obtained via crowdsourcing. The motivation for this work is to develop a testbed for image sequence description systems, where the task is to generate natural language descriptions for animated GIFs or video clips. To ensure a high quality dataset, we developed a series of novel quality controls to validate free-form text input from crowdworkers. We show that there is unambiguous association between visual content and natural language descriptions in our dataset, making it an ideal benchmark for the visual content captioning task. We perform extensive statistical analyses to compare our dataset to existing image and video description datasets. Next, we provide baseline results on the animated GIF description task, using three representative techniques: nearest neighbor, statistical machine translation, and recurrent neural networks. Finally, we show that models fine-tuned from our animated GIF description dataset can be helpful for automatic movie description.

Key facts. Cited by 124 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 17; 2022: 21; 2023: 20; 2024: 35; 2025: 23; 2026: 8.

124 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  3. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  4. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  5. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  6. 2024ShareGPT4Video: Improving Video Understanding and Generation with Better Captionscited by 386
  7. 2024LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understandingcited by 341
  8. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  9. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  10. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  11. 2025SmolVLM: Redefining small and efficient multimodal modelscited by 226
  12. 2023Video Understanding with Large Language Models: A Surveycited by 203
38 more
  1. 2025Unified Reward Model for Multimodal Understanding and Generationcited by 194
  2. 2023VAST: A Vision-Audio-Subtitle-Text Omni-Modality Foundation Model and Datasetcited by 168
  3. 2024Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMscited by 153
  4. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  5. 2024SlowFast-LLaVA: A Strong Training-Free Baseline for Video Large Language Modelscited by 123
  6. 2024An Image Grid Can Be Worth a Video: Zero-shot Video Question Answering Using a VLMcited by 97
  7. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  8. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  9. 2023ARTboldsymbol·V: Auto-Regressive Text-to-Video Generation with Diffusion Modelscited by 74
  10. 2024VideoVista: A Versatile Benchmark for Video Understanding and Reasoningcited by 72
  11. 2024TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Modelscited by 71
  12. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 68
  13. 2022From Representation to Reasoning: Towards both Evidence and Commonsense Reasoning for Video Question-Answeringcited by 51
  14. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 50
  15. 2023Reuse and Diffuse: Iterative Denoising for Text-to-Video Generationcited by 49
  16. 2024Vitron: A Unified Pixel-level Vision LLM for Understanding, Generating, Segmenting, Editingcited by 49
  17. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  18. 2023Zero-Shot Video Question Answering with Procedural Programscited by 37
  19. 2022Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrievalcited by 34
  20. 2023A Review of Deep Learning for Video Captioningcited by 28
  21. 2021A Bilingual, OpenWorld Video Text Dataset and End-to-end Video Text Spotter with Transformercited by 26
  22. 2023From Image to Language: A Critical Analysis of Visual Question Answering (VQA) Approaches, Challenges, and Opportunitiescited by 24
  23. 2023u-LLaVA: Unifying Multi-Modal Tasks via Large Language Modelcited by 24
  24. 2024AIGV-Assessor: Benchmarking and Evaluating the Perceptual Quality of Text-to-Video Generation with LMMcited by 24
  25. 2021Lightweight Attentional Feature Fusion: A New Baseline for Text-to-Video Retrievalcited by 23
  26. 2022SimVTP: Simple Video Text Pre-training with Masked Autoencoderscited by 21
  27. 2025NExT-OMNI: Towards Any-to-Any Omnimodal Foundation Models with Discrete Flow Matchingcited by 19
  28. 2022Modeling Motion with Multi-Modal Features for Text-Based Video Segmentationcited by 18
  29. 2024COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Trainingcited by 18
  30. 2024DAM: Dynamic Adapter Merging for Continual Video QA Learningcited by 18
  31. 2024What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interactioncited by 18
  32. 2021Question Answering Survey: Directions, Challenges, Datasets, Evaluation Matricescited by 17
  33. 2022Unifying Event Detection and Captioning as Sequence Generation via Pre-Trainingcited by 17
  34. 2023VaQuitA: Enhancing Alignment in LLM-Assisted Video Understandingcited by 17
  35. 2024WTS: A Pedestrian-Centric Traffic Video Dataset for Fine-grained Spatial-Temporal Understandingcited by 17
  36. 2024ViCaS: A Dataset for Combining Holistic and Pixel-level Video Understanding using Captions with Grounded Segmentationcited by 16
  37. 2025LoVR: A Benchmark for Long Video Retrieval in Multimodal Contextscited by 14
  38. 2025Bridging the Semantic-Action Gap in Visual Token Pruning for Efficient VLA Inferencecited by 14

50 most cited of 124 All 124 (JSON)