mlsanityData to 7 Oct 2026

CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoning

arXiv 1910.04744 ↗First posted 10 Oct 2019cs.CVcited by 137

Not on the map. Why?

Cited by 20 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Computer vision has undergone a dramatic revolution in performance, driven in large part through deep features trained on large-scale supervised datasets. However, much of these improvements have focused on static image analysis; video understanding has seen rather modest improvements. Even though new datasets and spatiotemporal models have been proposed, simple frame-by-frame classification methods often still remain competitive. We posit that current video datasets are plagued with implicit biases over scene and object structure that can dwarf variations in temporal structure. In this work, we build a video dataset with fully observable and controllable object and scene bias, and which truly requires spatiotemporal understanding in order to be solved. Our dataset, named CATER, is rendered synthetically using a library of standard 3D objects, and tests the ability to recognize compositions of object movements that require long-term reasoning. In addition to being a challenging dataset, CATER also provides a plethora of diagnostic tools to analyze modern spatiotemporal video architectures by being completely observable and controllable. Using CATER, we provide insights into some of the most recent state of the art deep video architectures.

Key facts. Cited by 137 papers in the map (data through 7 Oct 2026). First posted on arXiv 10 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 29; 2023: 24; 2024: 25; 2025: 18; 2026: 13.

137 papers.

Sort:

By area:

  1. 2024TempCompass: Do Video LLMs Really Understand Videos?cited by 352
  2. 2023Perception Test: A Diagnostic Benchmark for Multimodal Video Modelscited by 310
  3. 2021Conditional Object-Centric Learning from Videocited by 247
  4. 2024STAR: A Benchmark for Situated Reasoning in Real-World Videoscited by 213
  5. 2021Anticipative Video Transformercited by 197
  6. 2022Revisiting the "Video" in Video-Language Understandingcited by 167
  7. 2022SAVi++: Towards End-to-End Object-Centric Learning from Real-World Videoscited by 165
  8. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  9. 2021LanguageRefer: Spatial-Language Model for 3D Visual Groundingcited by 88
  10. 2021Make It Move: Controllable Image-to-Video Generation with Text Descriptionscited by 88
  11. 2024TRACE: Temporal Grounding Video LLM via Causal Event Modelingcited by 87
  12. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
38 more
  1. 2021SIMONe: View-Invariant, Temporally-Abstracted Object Representations via Unsupervised Video Decompositioncited by 69
  2. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  3. 2022CLIP-TSA: CLIP-Assisted Temporal Self-Attention for Weakly-Supervised Video Anomaly Detectioncited by 67
  4. 2022ComPhy: Compositional Physical Reasoning of Objects and Events from Videoscited by 61
  5. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  6. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  7. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  8. 2021Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learningcited by 53
  9. 2023Going Denser with Open-Vocabulary Part Segmentationcited by 51
  10. 2025IntPhys 2: Benchmarking Intuitive Physics Understanding In Complex Synthetic Environmentscited by 49
  11. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  12. 2022Discovering Objects that Can Movecited by 44
  13. 2022Guess What Moves: Unsupervised Video and Image Segmentation by Anticipating Motioncited by 42
  14. 2025Omnia de EgoTempo: Benchmarking Temporal Understanding of Multi-Modal LLMs in Egocentric Videoscited by 41
  15. 2023Benchmarks for Automated Commonsense Reasoning: A Surveycited by 40
  16. 2021ACRE: Abstract Causal REasoning Beyond Covariationcited by 30
  17. 2024How Good is my Video LMM? Complex Video Reasoning and Robustness Evaluation Suite for Video-LMMscited by 30
  18. 2022(2.5+1)D Spatio-Temporal Scene Graphs for Video Question Answeringcited by 29
  19. 2021Generative Video Transformer: Can Objects be the Words?cited by 24
  20. 2022Is an Object-Centric Video Representation Beneficial for Transfer?cited by 24
  21. 2024MECD: Unlocking Multi-Event Causal Discovery in Video Reasoningcited by 23
  22. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  23. 2025V-ReasonBench: Toward Unified Reasoning Benchmark Suite for Video Generation Modelscited by 22
  24. 2021Capturing the objects of vision with neural networkscited by 21
  25. 2022Learning What and Where: Disentangling Location and Identity Tracking Without Supervisioncited by 21
  26. 2024PhysGame: Uncovering Physical Commonsense Violations in Gameplay Videoscited by 21
  27. 2025An Empirical Study of Autoregressive Pre-training from Videoscited by 21
  28. 2021Transformers with Competitive Ensembles of Independent Mechanismscited by 19
  29. 2022Object Permanence Emerges in a Random Walk along Memorycited by 18
  30. 2024Compositional 4D Dynamic Scenes Understanding with Physics Priors for Video Question Answeringcited by 18
  31. 2022Revealing Occlusions with 4D Neural Fieldscited by 17
  32. 2023Shepherding Slots to Objects: Towards Stable and Robust Object-Centric Learningcited by 17
  33. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  34. 2024Slot State Space Modelscited by 17
  35. 2024Survey of Large Multimodal Model Datasets, Application Categories and Taxonomycited by 17
  36. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  37. 2022Measuring Compositional Consistency for Video Question Answeringcited by 15
  38. 2022Cross-modal Representation Learning for Zero-shot Action Recognitioncited by 15

50 most cited of 137 All 137 (JSON)