mlsanityData to 14 Oct 2026

HAKE: A Knowledge Engine Foundation for Human Activity Understanding

arXiv 2202.06851 ↗First posted 14 Feb 2022cs.CV cs.AI cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Human activity understanding is of widespread interest in artificial intelligence and spans diverse applications like health care and behavior analysis. Although there have been advances in deep learning, it remains challenging. The object recognition-like solutions usually try to map pixels to semantics directly, but activity patterns are much different from object patterns, thus hindering success. In this work, we propose a novel paradigm to reformulate this task in two stages: first mapping pixels to an intermediate space spanned by atomic activity primitives, then programming detected primitives with interpretable logic rules to infer semantics. To afford a representative primitive space, we build a knowledge base including 26+ M primitive labels and logic rules from human priors or automatic discovering. Our framework, the Human Activity Knowledge Engine (HAKE), exhibits superior generalization ability and performance upon canonical methods on challenging benchmarks. Code and data are available at http://hake-mvig.cn/.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 49 of its 79 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

49 of 79 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  6. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  7. 2018SlowFast Networks for Video Recognitioncited by 1,867
  8. 2014Two-Stream Convolutional Networks for Action Recognition in Videoscited by 1,068
  9. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  10. 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
  11. 2016Convolutional Two-Stream Network Fusion for Video Action Recognitioncited by 384
  12. 2015Neural Module Networkscited by 376
37 more
  1. 2019The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervisioncited by 372
  2. 2016RMPE: Regional Multi-person Pose Estimationcited by 340
  3. 2016Visual Relationship Detection with Language Priorscited by 334
  4. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  5. 2017Learning to Detect Human-Object Interactionscited by 240
  6. 2018Long-Term Feature Banks for Detailed Video Understandingcited by 239
  7. 2017Detecting and Recognizing Human-Object Interactionscited by 199
  8. 2015Visual Semantic Role Labelingcited by 190
  9. 2021HOTR: End-to-End Human-Object Interaction Detection with Transformerscited by 183
  10. 2018Learning Human-Object Interactions by Graph Parsing Neural Networkscited by 176
  11. 2017End-to-End Differentiable Provingcited by 161
  12. 2019Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoningcited by 151
  13. 2021QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Informationcited by 147
  14. 2019Neural Logic Machinescited by 145
  15. 2019PPDM: Parallel Point Detection and Matching for Real-time Human-Object Interaction Detectioncited by 143
  16. 2020Visual Compositional Learning for Human-Object Interaction Detectioncited by 130
  17. 2018iCAN: Instance-Centric Attention Network for Human-Object Interaction Detectioncited by 127
  18. 2019Learning by Abstraction: The Neural State Machinecited by 113
  19. 2019AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videoscited by 105
  20. 2020Detailed 2D-3D Joint Representation for Human-Object Interactioncited by 93
  21. 2019Pose-aware Multi-level Feature Network for Human Object Interaction Detectioncited by 88
  22. 2020HOI Analysis: Integrating and Decomposing Human-Object Interactioncited by 81
  23. 2020PaStaNet: Toward Human Activity Knowledge Enginecited by 80
  24. 2020Detecting Human-Object Interactions with Action Co-occurrence Priorscited by 72
  25. 2019Learning to Infer and Execute 3D Shape Programscited by 68
  26. 2019Visually Grounded Neural Syntax Acquisitioncited by 48
  27. 2015Human Action Recognition using Factorized Spatio-Temporal Convolutional Networkscited by 40
  28. 2020Neural Logic Reasoningcited by 38
  29. 2018Pairwise Body-Part Attention for Recognizing Human-Object Interactionscited by 34
  30. 2016Single Image Action Recognition using Semantic Body Part Actionscited by 14
  31. 2021Hierarchical Motion Understanding via Motion Programscited by 14
  32. 2016Beyond Holistic Object Recognition: Enriching Image Understanding with Part Statescited by 13
  33. 2014Actions and Attributes from Wholes and Partscited by 12
  34. 2019Program-Guided Image Manipulatorscited by 12
  35. 2020Multi-Plane Program Induction with 3D Box Priorscited by 10
  36. 2017Care about you: towards large-scale human-centric visual relationship detectioncited by 6
  37. 2018Detecting unseen visual relations using analogiescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 8; 2024: 8; 2025: 1; 2026: 3.

28 papers.

Sort:

By area:

  1. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
  2. 2022HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modelingcited by 117
  3. 2022Generative Action Description Prompts for Skeleton-based Action Recognitioncited by 43
  4. 2022Interactiveness Field in Human-Object Interactionscited by 40
  5. 2022Mining Cross-Person Cues for Body-Part Interactiveness Learning in HOI Detectioncited by 34
  6. 2022Constructing Balance from Imbalance for Long-tailed Image Recognitioncited by 25
  7. 2024Towards Multimodal In-Context Learning for Vision & Language Modelscited by 25
  8. 2023Can Linguistic Knowledge Improve Multimodal Alignment in Vision-Language Pretraining?cited by 21
  9. 2024Real-time Transformer-based Open-Vocabulary Detection with Efficient Fusion Headcited by 20
  10. 2023From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understandingcited by 19
  11. 2022PACS: A Dataset for Physical Audiovisual CommonSense Reasoningcited by 18
  12. 2023Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoningcited by 16
16 more
  1. 2023Disentangled Counterfactual Learning for Physical Audiovisual Commonsense Reasoningcited by 13
  2. 2025Reconstructing In-the-Wild Open-Vocabulary Human-Object Interactionscited by 13
  3. 2022Beyond Object Recognition: A New Benchmark towards Object Concept Learningcited by 11
  4. 2022Discovering A Variety of Objects in Spatio-Temporal Human-Object Interactionscited by 6
  5. 2024Take A Step Back: Rethinking the Two Stages in Visual Reasoningcited by 4
  6. 2023Enhancing HOI Detection with Contextual Cues from Large Vision-Language Modelscited by 3
  7. 2023BPJDet: Extended Object Representation for Generic Body-Part Joint Detectioncited by 2
  8. 2024Semi-Supervised Unconstrained Head Pose Estimation in the Wildcited by 2
  9. 2024ELSA: Evaluating Localization of Social Activities in Urban Streets using Open-Vocabulary Detectioncited by 2
  10. 2023Open-Vocabulary Video Relation Extractioncited by 1
  11. 2024KALE-LM-Chem: Vision and Practice Toward an AI Brain for Chemistrycited by 1
  12. 2024Homogeneous Dynamics Space for Heterogeneous Humanscited by 1
  13. 2026Neurosymbolic Framework for Concept-Driven Logical Reasoning in Skeleton-Based Human Action Recognitioncited by 1
  14. 2024Verb Mirage: Unveiling and Assessing Verb Concept Hallucinations in Multimodal Large Language Modelscited by 0
  15. 2026Beyond Static Vision: Scene Dynamic Field Unlocks Intuitive Physics Understanding in Multi-modal Large Language Modelscited by 0
  16. 2026A Neurosymbolic Framework for Interpretable Skeleton-Based Seizure Detection via Concept-Driven Logical Reasoningcited by 0