mlsanityData to 7 Oct 2026

The "something something" video database for learning and evaluating visual common sense

arXiv 1706.04261 ↗First posted 13 Jun 2017cs.CVcited by 1,092

Not on the map. Why?

Cited by 84 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Neural networks trained on datasets such as ImageNet have led to major advances in visual object classification. One obstacle that prevents networks from reasoning more deeply about complex scenes and situations, and from integrating visual knowledge with natural language, like humans do, is their lack of common sense knowledge about the physical world. Videos, unlike still images, contain a wealth of detailed information about the physical world. However, most labelled video datasets represent high-level concepts rather than detailed physical aspects about actions and scenes. In this work, we describe our ongoing collection of the "something-something" database of video prediction tasks whose solutions require a common sense understanding of the depicted situation. The database currently contains more than 100,000 videos across 174 classes, which are defined as caption-templates. We also describe the challenges in crowd-sourcing this data at scale.

Key facts. Cited by 1,092 papers in the map (data through 7 Oct 2026). First posted on arXiv 13 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 130; 2022: 178; 2023: 181; 2024: 222; 2025: 222; 2026: 159.

1,092 papers.

Sort:

By area:

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2021ViViT: A Video Vision Transformercited by 1,610
  3. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  4. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,530
  5. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  6. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  7. 2021Video Swin Transformercited by 1,090
  8. 2023MVBench: A Comprehensive Multi-modal Video Understanding Benchmarkcited by 1,086
  9. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  10. 2021Multiscale Vision Transformerscited by 896
  11. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  12. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
38 more
  1. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  2. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  3. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  4. 2023EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understandingcited by 666
  5. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  6. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  7. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 573
  8. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 550
  9. 2021MViTv2: Improved Multiscale Vision Transformers for Classification and Detectioncited by 516
  10. 2023VideoMAE V2: Scaling Video Masked Autoencoders with Dual Maskingcited by 516
  11. 2024Revisiting Feature Prediction for Learning Visual Representations from Videocited by 490
  12. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  13. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 479
  14. 2023Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectivescited by 477
  15. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  16. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 441
  17. 2023Learning Interactive Real-World Simulatorscited by 440
  18. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  19. 2022Masked Autoencoders As Spatiotemporal Learnerscited by 428
  20. 2024OmniGen: Unified Image Generationcited by 388
  21. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  22. 2022MAGVIT: Masked Generative Video Transformercited by 341
  23. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  24. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 320
  25. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  26. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 309
  27. 2024Latent Action Pretraining from Videoscited by 302
  28. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  29. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  30. 2023Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistlescited by 287
  31. 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
  32. 2024Video Prediction Policy: A Generalist Robot Policy with Predictive Visual Representationscited by 279
  33. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  34. 2021H2O: Two Hands Manipulating Objects for First Person Interaction Recognitioncited by 251
  35. 2022Masked Visual Pre-training for Motor Controlcited by 246
  36. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  37. 2021Keeping Your Eye on the Ball: Trajectory Attention in Video Transformerscited by 224
  38. 2025Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasetscited by 218

50 most cited of 1,092 All 1,092 (JSON)