# ReStory: VLM-augmentation of Social Human-Robot Interaction Datasets

arXiv: https://arxiv.org/abs/2412.20826 · HTML: https://mlsanity.com/paper/2412.20826/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Dec 2024.

## Built on by (most cited first, 1 of 1)

- [Understanding Pedestrian Gesture Misrecognition: Insights from Vision-Language Model Reasoning](https://mlsanity.com/paper/2508.06801/) (2025)

## Abstract (arXiv)

Internet-scaled datasets are a luxury for human-robot interaction (HRI) researchers, as collecting natural interaction data in the wild is time-consuming and logistically challenging. The problem is exacerbated by robots' different form factors and interaction modalities. Inspired by recent work on ethnomethodological and conversation analysis (EMCA) in the domain of HRI, we propose ReStory, a method that has the potential to augment existing in-the-wild human-robot interaction datasets leveraging Vision Language Models. While still requiring human supervision, ReStory is capable of synthesizing human-interpretable interaction scenarios in the form of storyboards. We hope our proposed approach provides HRI researchers and interaction designers with a new angle to utilizing their valuable and scarce data.