# Automatic Interaction and Activity Recognition from Videos of Human Manual Demonstrations with Application to Anomaly Detection

arXiv: https://arxiv.org/abs/2304.09789 · HTML: https://mlsanity.com/paper/2304.09789/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 35 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 5 of 5)

- [A Comprehensive Survey of Scene Graphs: Generation and Application](https://mlsanity.com/paper/2104.01111/) (2021)
- [Learning Object-Action Relations from Bimanual Human Demonstration Using Graph Networks](https://mlsanity.com/paper/1908.08391/) (2019)
- [Motion Guided Attention Fusion to Recognize Interactions from Videos](https://mlsanity.com/paper/2104.00646/) (2021)
- [Pick the Right Co-Worker: Online Assessment of Cognitive Ergonomics in Human-Robot Collaborative Assembly](https://mlsanity.com/paper/2207.03779/) (2022)
- [Reasoning on Grasp-Action Affordances](https://mlsanity.com/paper/1905.10610/) (2019)

## Built on by (most cited first, 3 of 3)

- [From Objects to Events: Unlocking Complex Visual Understanding in Object Detectors via LLM-guided Symbolic Reasoning](https://mlsanity.com/paper/2502.05843/) (2025)
- [Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaboration](https://mlsanity.com/paper/2511.08732/) (2025)
- [Lightweight Facial Landmark Detection in Thermal Images via Multi-Level Cross-Modal Knowledge Transfer](https://mlsanity.com/paper/2510.11128/) (2025)

## Abstract (arXiv)

This paper presents a new method to describe spatio-temporal relations between objects and hands, to recognize both interactions and activities within video demonstrations of manual tasks. The approach exploits Scene Graphs to extract key interaction features from image sequences while simultaneously encoding motion patterns and context. Additionally, the method introduces event-based automatic video segmentation and clustering, which allow for the grouping of similar events and detect if a monitored activity is executed correctly. The effectiveness of the approach was demonstrated in two multi-subject experiments, showing the ability to recognize and cluster hand-object and object-object interactions without prior knowledge of the activity, as well as matching the same activity performed by different subjects.