mlsanityData to 7 Oct 2026

Hollywood in Homes: Crowdsourcing Data Collection for Activity Understanding

arXiv 1604.01753 ↗First posted 6 Apr 2016cs.CVcited by 535

Not on the map. Why?

Cited by 48 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computer vision has a great potential to help our daily lives by searching for lost keys, watering flowers or reminding us to take a pill. To succeed with such tasks, computer vision methods need to be trained from real and diverse examples of our daily dynamic scenes. While most of such scenes are not particularly exciting, they typically do not appear on YouTube, in movies or TV broadcasts. So how do we collect sufficiently many diverse but boring samples representing our lives? We propose a novel Hollywood in Homes approach to collect such data. Instead of shooting videos in the lab, we ensure diversity by distributing and crowdsourcing the whole process of video creation from script writing to video recording and annotation. Following this procedure we collect a new dataset, Charades, with hundreds of people recording videos in their own homes, acting out casual everyday activities. The dataset is composed of 9,848 annotated videos with an average length of 30 seconds, showing activities of 267 people from three continents. Each video is annotated by multiple free-text descriptions, action labels, action intervals and classes of interacted objects. In total, Charades provides 27,847 video descriptions, 66,500 temporally localized intervals for 157 action classes and 41,104 labels for 46 object classes. Using this rich data, we evaluate and provide baseline results for several tasks including action recognition and automatic description generation. We believe that the realism, diversity, and casual nature of this dataset will present unique challenges and new opportunities for computer vision community.

Key facts. Cited by 535 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Apr 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 110; 2022: 96; 2023: 99; 2024: 91; 2025: 91; 2026: 48.

535 papers.

Sort:

By area:

  1. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2021Transformers in Vision: A Surveycited by 1,009
  4. 2021Multiscale Vision Transformerscited by 896
  5. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  6. 2022DiffusionDet: Diffusion Model for Object Detectioncited by 354
  7. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 317
  8. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  9. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  10. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  11. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  12. 2021A Large-Scale Study on Unsupervised Spatiotemporal Representation Learningcited by 214
38 more
  1. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  2. 2023Video Understanding with Large Language Models: A Surveycited by 203
  3. 2021Towards Long-Form Video Understandingcited by 146
  4. 2024Tarsier: Recipes for Training and Evaluating Large Video Description Modelscited by 140
  5. 2022EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relationscited by 139
  6. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 138
  7. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 138
  8. 2024AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmarkcited by 137
  9. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  10. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 135
  11. 2024Kangaroo: A Powerful Video-Language Model Supporting Long-context Video Inputcited by 132
  12. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  13. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  14. 2024LITA: Language Instructed Temporal-Localization Assistantcited by 124
  15. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  16. 2021VidTr: Video Transformer Without Convolutionscited by 112
  17. 2025VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documentscited by 110
  18. 2021Context-aware Biaffine Localizing Network for Temporal Sentence Groundingcited by 107
  19. 2021MAD: A Scalable Dataset for Language Grounding in Videos from Movie Audio Descriptionscited by 107
  20. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  21. 2022Animal Kingdom: A Large and Diverse Dataset for Animal Behavior Understandingcited by 102
  22. 2023SEED-Bench-2: Benchmarking Multimodal Large Language Modelscited by 96
  23. 2021Spatial-Temporal Transformer for Dynamic Scene Graph Generationcited by 95
  24. 2021Interventional Video Grounding with Dual Contrastive Learningcited by 94
  25. 2021Negative Sample Matters: A Renaissance of Metric Learning for Temporal Groundingcited by 90
  26. 2021An Image is Worth 16x16 Words, What is a Video Worth?cited by 87
  27. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  28. 2021Deconfounded Video Moment Retrieval with Causal Interventioncited by 84
  29. 2022Temporal Alignment Networks for Long-term Videocited by 84
  30. 2021MultiSports: A Multi-Person Video Dataset of Spatio-Temporally Localized Sports Actionscited by 82
  31. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  32. 2022VideoCoCa: Video-Text Modeling with Zero-Shot Transfer from Contrastive Captionerscited by 69
  33. 2021Temporal Query Networks for Fine-grained Video Understandingcited by 68
  34. 2024E.T. Bench: Towards Open-Ended Event-Level Video-Language Understandingcited by 68
  35. 2022BURST: A Benchmark for Unifying Object Recognition, Segmentation and Tracking in Videocited by 66
  36. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  37. 2021SiamMOT: Siamese Multi-Object Trackingcited by 62
  38. 2021Visual Semantic Role Labeling for Video Understandingcited by 60

50 most cited of 535 All 535 (JSON)