mlsanityData to 7 Oct 2026

Disentangled Action Recognition with Knowledge Bases

arXiv 2207.01708 ↗First posted 4 Jul 2022cs.CV cs.AI cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Action in video usually involves the interaction of human with objects. Action labels are typically composed of various combinations of verbs and nouns, but we may not have training data for all possible combinations. In this paper, we aim to improve the generalization ability of the compositional action recognition model to novel verbs or novel nouns that are unseen during training time, by leveraging the power of knowledge graphs. Previous work utilizes verb-noun compositional action nodes in the knowledge graph, making it inefficient to scale since the number of compositional action nodes grows quadratically with respect to the number of verbs and nouns. To address this issue, we propose our approach: Disentangled Action Recognition with Knowledge-bases (DARK), which leverages the inherent compositionality of actions. DARK trains a factorized model by first extracting disentangled feature representations for verbs and nouns, and then predicting classification weights using relations in external knowledge graphs. The type constraint between verb and noun is extracted from external knowledge bases and finally applied when composing actions. DARK has better scalability in the number of objects and verbs, and achieves state-of-the-art performance on the Charades dataset. We further propose a new benchmark split based on the Epic-kitchen dataset which is an order of magnitude bigger in the numbers of classes and samples, and benchmark various models on this benchmark.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 33 of its 57 references that are in the map. First posted on arXiv 4 Jul 2022.

Builds on

33 of 57 references.

References · 33

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  3. 2015Fast R-CNNcited by 2,930
  4. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  5. 2017The Kinetics Human Action Video Datasetcited by 1,894
  6. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  7. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  8. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  9. 2018Scaling Egocentric Vision: The EPIC-KITCHENS Datasetcited by 789
  10. 2017Zero-Shot Learning -- A Comprehensive Evaluation of the Good, the Bad and the Uglycited by 536
  11. 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
  12. 2016Visual Relationship Detection with Language Priorscited by 333
21 more
  1. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  2. 2015A Review of Relational Machine Learning for Knowledge Graphscited by 264
  3. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  4. 2017Hide-and-Seek: Forcing a Network to be Meticulous for Weakly-supervised Object and Action Localizationcited by 207
  5. 2019Cross-task weakly supervised learning from instructional videoscited by 199
  6. 2015Label-Embedding for Image Classificationcited by 164
  7. 2018Zero-shot Recognition via Semantic Embeddings and Knowledge Graphscited by 155
  8. 2016Learning a Deep Embedding Model for Zero-Shot Learningcited by 153
  9. 2016An Empirical Study and Analysis of Generalized Zero-Shot Learning for Object Recognition in the Wildcited by 146
  10. 2013Zero-Shot Learning by Convex Combination of Semantic Embeddingscited by 135
  11. 2019Task-Driven Modular Networks for Zero-Shot Compositional Learningcited by 121
  12. 2019Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognitioncited by 117
  13. 2019Something-Else: Compositional Action Recognition with Spatial-Temporal Interaction Networkscited by 112
  14. 2018Variational Discriminator Bottleneck: Improving Imitation Learning, Inverse RL, and GANs by Constraining Information Flowcited by 103
  15. 2018Rethinking Knowledge Graph Propagation for Zero-Shot Learningcited by 87
  16. 2017Multi-Label Zero-Shot Learning with Structured Knowledge Graphscited by 72
  17. 2017Towards Context-aware Interaction Recognitioncited by 20
  18. 2015Semantic Embedding Space for Zero-Shot Action Recognitioncited by 19
  19. 2020Zero-Shot Learning with Common Sense Knowledge Graphscited by 17
  20. 2018Polarity Loss for Zero-shot Object Detectioncited by 14
  21. 2020Generative Adversarial Zero-shot Learning via Knowledge Graphscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 0; 2025: 1; 2026: 2.

6 papers.

  1. 2023NEUCORE: Neural Concept Reasoning for Composed Image Retrievalcited by 8
  2. 2025Can masking background and object reduce static bias for zero-shot action recognition?cited by 6
  3. 2022REST: REtrieve & Self-Train for generative action recognitioncited by 1
  4. 2023Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagationcited by 1
  5. 2026Polyphony: Diffusion-based Dual-Hand Action Segmentation with Alternating Vision Transformer and Semantic Conditioningcited by 1
  6. 2026Why Can't I Open My Drawer? Mitigating Object-Driven Shortcuts in Zero-Shot Compositional Action Recognitioncited by 0