mlsanityData to 7 Oct 2026

Moments in Time Dataset: one million videos for event understanding

arXiv 1801.03150 ↗First posted 9 Jan 2018cs.CV cs.AIcited by 215

Not on the map. Why?

Abstract (arXiv)

We present the Moments in Time Dataset, a large-scale human-annotated collection of one million short videos corresponding to dynamic events unfolding within three seconds. Modeling the spatial-audio-temporal dynamics even for actions occurring in 3 second videos poses many challenges: meaningful events do not include only people, but also objects, animals, and natural phenomena; visual and auditory events can be symmetrical in time ("opening" is "closing" in reverse), and either transient or sustained. We describe the annotation process of our dataset (each video is tagged with one action or activity label among 339 different classes), analyze its scale and diversity in comparison to other large-scale video datasets for action recognition, and report results of several baseline models addressing separately, and jointly, three modalities: spatial, temporal and auditory. The Moments in Time dataset, designed to have a large coverage and diversity of events in both visual and auditory modalities, can serve as a new challenge to develop models that scale to the level of complexity and abstract reasoning that a human processes on a daily basis.

Key facts. Cited by 215 papers in the map (data through 7 Oct 2026). First posted on arXiv 9 Jan 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 45; 2022: 52; 2023: 44; 2024: 31; 2025: 28; 2026: 15.

215 papers.

Sort:

By area:

  1. 2021ViViT: A Video Vision Transformercited by 1,610
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  4. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  5. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  6. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  7. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  8. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  9. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  10. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  11. 2021Video Transformer Networkcited by 271
  12. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
38 more
  1. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  2. 2023Unmasked Teacher: Towards Training-Efficient Video Foundation Modelscited by 199
  3. 2022Multiview Transformers for Video Recognitioncited by 148
  4. 2021MoViNets: Mobile Video Networks for Efficient Video Recognitioncited by 135
  5. 2021TokenLearner: What Can 8 Learned Tokens Do for Images and Videos?cited by 127
  6. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 124
  7. 2024OmniTokenizer: A Joint Image-Video Tokenizer for Visual Generationcited by 117
  8. 2022UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormercited by 116
  9. 2022HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modelingcited by 115
  10. 2022FineDiving: A Fine-grained Dataset for Procedure-aware Action Quality Assessmentcited by 106
  11. 2024VideoPrism: A Foundational Visual Encoder for Video Understandingcited by 103
  12. 2021Pre-training without Natural Imagescited by 100
  13. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  14. 2022Temporal Action Segmentation: An Analysis of Modern Techniquescited by 85
  15. 2022Source-Free Unsupervised Domain Adaptation: A Surveycited by 79
  16. 2023Verbs in Action: Improving verb understanding in video-language modelscited by 76
  17. 2021ACTION-Net: Multipath Excitation for Action Recognitioncited by 63
  18. 2021iMiGUE: An Identity-free Video Dataset for Micro-Gesture Understanding and Emotion Analysiscited by 63
  19. 2021Visual Semantic Role Labeling for Video Understandingcited by 60
  20. 2021FineAction: A Fine-Grained Video Dataset for Temporal Action Localizationcited by 60
  21. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  22. 2021ROAD: The ROad event Awareness Dataset for Autonomous Drivingcited by 54
  23. 2021Spoken Moments: Learning Joint Audio-Visual Representations from Video Descriptionscited by 53
  24. 2023FedMultimodal: A Benchmark For Multimodal Federated Learningcited by 51
  25. 2021ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learningcited by 50
  26. 2021AdaFuse: Adaptive Temporal Fusion Network for Efficient Action Recognitioncited by 48
  27. 2022HARDVS: Revisiting Human Activity Recognition with Dynamic Vision Sensorscited by 46
  28. 2021A Hybrid Attention Mechanism for Weakly-Supervised Temporal Action Localizationcited by 45
  29. 2024Consent in Crisis: The Rapid Decline of the AI Data Commonscited by 45
  30. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  31. 2023MAtch, eXpand and Improve: Unsupervised Finetuning for Zero-Shot Action Recognition with Language Knowledgecited by 43
  32. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  33. 2021Co-training Transformer with Videos and Images Improves Action Recognitioncited by 41
  34. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  35. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  36. 2021The SARAS Endoscopic Surgeon Action Detection (ESAD) dataset: Challenges and methodscited by 37
  37. 2024Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectivescited by 37
  38. 2021Dynamic Network Quantization for Efficient Video Inferencecited by 36

50 most cited of 215 All 215 (JSON)