# Describing Common Human Visual Actions in Images

arXiv: https://arxiv.org/abs/1506.02203 · HTML: https://mlsanity.com/paper/1506.02203/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Jun 2015.

## Built on by (most cited first, 12 of 12)

- [Measuring Representational Harms in Image Captioning](https://mlsanity.com/paper/2206.07173/) (2022)
- [Bongard-HOI: Benchmarking Few-Shot Visual Reasoning for Human-Object Interactions](https://mlsanity.com/paper/2205.13803/) (2022)
- [JRDB-Social: A Multifaceted Robotic Dataset for Understanding of Context and Dynamics of Human Interactions Within Social Groups](https://mlsanity.com/paper/2404.04458/) (2024)
- [Ambiguous Images With Human Judgments for Robust Visual Event Classification](https://mlsanity.com/paper/2210.03102/) (2022)
- [Visual Question Answering: which investigated applications?](https://mlsanity.com/paper/2103.02937/) (2021)
- [Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?](https://mlsanity.com/paper/2210.12079/) (2022)
- [Learning Human-Human Interactions in Images from Weak Textual Supervision](https://mlsanity.com/paper/2304.14104/) (2023)
- [Seeing vs. Believing: Evaluating the Language Bias of Open-Source MLLMs in Counter-Intuitive Scenes](https://mlsanity.com/paper/2601.07737/) (2026)
- [NeSy4VRD: A Multifaceted Resource for Neurosymbolic AI Research using Knowledge Graphs in Visual Relationship Detection](https://mlsanity.com/paper/2305.13258/) (2023)
- [Human Action Co-occurrence in Lifestyle Vlogs using Graph Link Prediction](https://mlsanity.com/paper/2309.06219/) (2023)
- [When concept-based XAI is imprecise: Do people distinguish between generalisations and misrepresentations?](https://mlsanity.com/paper/2506.17936/) (2025)
- [Towards Robust Evaluation of Visual Activity Recognition: Resolving Verb Ambiguity with Sense Clustering](https://mlsanity.com/paper/2508.04945/) (2025)

## Abstract (arXiv)

Which common human actions and interactions are recognizable in monocular still images? Which involve objects and/or other people? How many is a person performing at a time? We address these questions by exploring the actions and interactions that are detectable in the images of the MS COCO dataset. We make two main contributions. First, a list of 140 common `visual actions', obtained by analyzing the largest on-line verb lexicon currently available for English (VerbNet) and human sentences used to describe images in MS COCO. Second, a complete set of annotations for those `visual actions', composed of subject-object and associated verb, which we call COCO-a (a for `actions'). COCO-a is larger than existing action datasets in terms of number of actions and instances of these actions, and is unique because it is data-driven, rather than experimenter-biased. Other unique features are that it is exhaustive, and that all subjects and objects are localized. A statistical analysis of the accuracy of our annotations and of each action, interaction and subject-object combination is provided.