mlsanityData to 14 Oct 2026

Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networks

arXiv 1912.09930 ↗First posted 20 Dec 2019cs.CVcited by 114

Not on the map. Why?

Abstract (arXiv)

Human action is naturally compositional: humans can easily recognize and perform actions with objects that are different from those used in training demonstrations. In this paper, we study the compositionality of action by looking into the dynamics of subject-object interactions. We propose a novel model which can explicitly reason about the geometric relations between constituent objects and an agent performing an action. To train our model, we collect dense object box annotations on the Something-Something dataset. We propose a novel compositional action recognition task where the training combinations of verbs and nouns do not overlap with the test set. The novel aspects of our model are applicable to activities with prominent object interaction dynamics and to objects which can be tracked using state-of-the-art approaches; for activities without clearly defined spatial object-agent interactions, we rely on baseline scene-level spatio-temporal representations. We show the effectiveness of our approach not only on the proposed compositional action recognition task, but also in a few-shot compositional setting which requires the model to generalize across both object appearance and action category.

Key facts. Cited by 114 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 21; 2022: 28; 2023: 27; 2024: 16; 2025: 11; 2026: 11.

114 papers.

Sort:

By area:

  1. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 164
  2. 2025Magma: A Foundation Model for Multimodal AI Agentscited by 139
  3. 2024TACO: Benchmarking Generalizable Bimanual Tool-ACtion-Object Understandingcited by 88
  4. 2021ArtiBoost: Boosting Articulated 3D Hand-Object Pose Estimation via Online Exploration and Synthesiscited by 76
  5. 2021Object-Region Video Transformerscited by 73
  6. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  7. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  8. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  9. 2021ACTION-Net: Multipath Excitation for Action Recognitioncited by 63
  10. 2022Compositional Temporal Grounding with Structured Variational Cross-Graph Correspondence Learningcited by 59
  11. 2022Graph Inverse Reinforcement Learning from Diverse Videoscited by 46
  12. 2023Implicit Temporal Modeling with Learnable Alignment for Video Recognitioncited by 46
38 more
  1. 2021Unified Graph Structured Models for Video Understandingcited by 38
  2. 2022Interaction Region Visual Transformer for Egocentric Action Anticipationcited by 37
  3. 2022Latent Graph Representations for Critical View of Safety Assessmentcited by 37
  4. 2023Learning Object State Changes in Videos: An Open-World Perspectivecited by 36
  5. 2024Rank2Reward: Learning Shaped Reward Functions from Passive Videocited by 32
  6. 2021Learning Self-Similarity in Space and Time as Generalized Motion for Video Action Recognitioncited by 31
  7. 2023Helping Hands: An Object-Aware Ego-Centric Video Recognition Modelcited by 31
  8. 2022Probabilistic Compositional Embeddings for Multimodal Image Retrievalcited by 29
  9. 2022BasicTAD: an Astounding RGB-Only Baseline for Temporal Action Detectioncited by 28
  10. 2022Holistic Interaction Transformer Network for Action Detectioncited by 28
  11. 2023Multimodal Distillation for Egocentric Action Recognitioncited by 28
  12. 2025Exo2Ego: Exocentric Knowledge Guided MLLM for Egocentric Video Understandingcited by 25
  13. 2022Is an Object-Centric Video Representation Beneficial for Transfer?cited by 24
  14. 2023Object-centric Video Representation for Long-term Action Anticipationcited by 24
  15. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  16. 2023How can objects help action recognition?cited by 21
  17. 2021Video-Text Pre-training with Learned Regionscited by 20
  18. 2021Revisiting spatio-temporal layouts for compositional action recognitioncited by 19
  19. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  20. 2023Variational Cross-Graph Reasoning and Adaptive Structured Semantics Learning for Compositional Temporal Groundingcited by 19
  21. 2024Controlling the World by Sleight of Handcited by 19
  22. 2024What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interactioncited by 18
  23. 2022Geometric Features Informed Multi-person Human-object Interaction Recognition in Videoscited by 17
  24. 2021EAN: Event Adaptive Network for Enhanced Action Recognitioncited by 16
  25. 2023Leveraging Next-Active Objects for Context-Aware Anticipation in Egocentric Videoscited by 16
  26. 2022Discovering Human-Object Interaction Concepts via Self-Compositional Learningcited by 15
  27. 2022Cross-modal Representation Learning for Zero-shot Action Recognitioncited by 15
  28. 2023EgoPCA: A New Framework for Egocentric Hand-Object Interaction Understandingcited by 15
  29. 2022PromptonomyViT: Multi-Task Prompt Learning Improves Video Transformers using Synthetic Scene Datacited by 14
  30. 2023Look, Remember and Reason: Grounded reasoning in videos with language modelscited by 14
  31. 2023Encoding Surgical Videos as Latent Spatiotemporal Graphs for Object and Anatomy-Driven Reasoningcited by 14
  32. 2024Multimodal Cross-Domain Few-Shot Learning for Egocentric Action Recognitioncited by 14
  33. 2021Composable Augmentation Encoding for Video Representation Learningcited by 13
  34. 2021Skeleton-Based Mutually Assisted Interacted Object Localization and Human Action Recognitioncited by 13
  35. 2023LASER: A Neuro-Symbolic Framework for Learning Spatial-Temporal Scene Graphs with Weak Supervisioncited by 13
  36. 2023Chop & Learn: Recognizing and Generating Object-State Compositionscited by 13
  37. 2021Spatio-Temporal Interaction Graph Parsing Networks for Human-Object Interaction Recognitioncited by 12
  38. 2022TFCNet: Temporal Fully Connected Networks for Static Unbiased Temporal Reasoningcited by 12

50 most cited of 114 All 114 (JSON)