# Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering

arXiv: https://arxiv.org/abs/1604.04808 · HTML: https://mlsanity.com/paper/1604.04808/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Apr 2016.

## Built on by (most cited first, 15 of 15)

- [Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos](https://mlsanity.com/paper/2109.03223/) (2021)
- [Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videos](https://mlsanity.com/paper/2507.15597/) (2025)
- [CholecTriplet2021: A benchmark challenge for surgical action triplet recognition](https://mlsanity.com/paper/2204.04746/) (2022)
- [CholecTriplet2022: Show me a tool and tell me the triplet -- an endoscopic vision challenge for surgical action triplet detection](https://mlsanity.com/paper/2302.06294/) (2023)
- [From Isolated Islands to Pangea: Unifying Semantic Space for Human Action Understanding](https://mlsanity.com/paper/2304.00553/) (2023)
- [Geometric Features Informed Multi-person Human-object Interaction Recognition in Videos](https://mlsanity.com/paper/2207.09425/) (2022)
- [Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning](https://mlsanity.com/paper/2311.17365/) (2023)
- [RelViT: Concept-guided Vision Transformer for Visual Relational Reasoning](https://mlsanity.com/paper/2204.11167/) (2022)
- [Attend and Guide (AG-Net): A Keypoints-driven Attention-based Deep Network for Image Recognition](https://mlsanity.com/paper/2110.12183/) (2021)
- [The Overlooked Classifier in Human-Object Interaction Recognition](https://mlsanity.com/paper/2112.06392/) (2021)
- [Hypergraph-Transformer (HGT) for Interactive Event Prediction in Laparoscopic and Robotic Surgery](https://mlsanity.com/paper/2402.01974/) (2024)
- [From Category to Scenery: An End-to-End Framework for Multi-Person Human-Object Interaction Recognition in Videos](https://mlsanity.com/paper/2407.00917/) (2024)
- [Is Object Detection Necessary for Human-Object Interaction Recognition?](https://mlsanity.com/paper/2107.13083/) (2021)
- [Transferable Interactiveness Knowledge for Human-Object Interaction Detection](https://mlsanity.com/paper/2101.10292/) (2021)
- [The Overlooked Classifier in Human-Object Interaction Recognition](https://mlsanity.com/paper/2203.05676/) (2022)

## Abstract (arXiv)

This paper proposes deep convolutional network models that utilize local and global context to make human activity label predictions in still images, achieving state-of-the-art performance on two recent datasets with hundreds of labels each. We use multiple instance learning to handle the lack of supervision on the level of individual person instances, and weighted loss to handle unbalanced training data. Further, we show how specialized features trained on these datasets can be used to improve accuracy on the Visual Question Answering (VQA) task, in the form of multiple choice fill-in-the-blank questions (Visual Madlibs). Specifically, we tackle two types of questions on person activity and person-object relationship and show improvements over generic features trained on the ImageNet classification task.