# Gesture-aware Interactive Machine Teaching with In-situ Object Annotations

arXiv: https://arxiv.org/abs/2208.01211 · HTML: https://mlsanity.com/paper/2208.01211/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Aug 2022.

## Built on by (most cited first, 6 of 6)

- [Next Steps for Human-Centered Generative AI: A Technical Perspective](https://mlsanity.com/paper/2306.15774/) (2023)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agents](https://mlsanity.com/paper/2509.12049/) (2025)
- [InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMs](https://mlsanity.com/paper/2312.09672/) (2023)
- [Low-Barrier Dataset Collection with Real Human Body for Interactive Per-Garment Virtual Try-On](https://mlsanity.com/paper/2506.10468/) (2025)
- [Physical Annotation for Automated Optical Inspection: A Concept for In-Situ, Pointer-Based Training Data Generation](https://mlsanity.com/paper/2506.05026/) (2025)

## Abstract (arXiv)

Interactive Machine Teaching (IMT) systems allow non-experts to easily create Machine Learning (ML) models. However, existing vision-based IMT systems either ignore annotations on the objects of interest or require users to annotate in a post-hoc manner. Without the annotations on objects, the model may misinterpret the objects using unrelated features. Post-hoc annotations cause additional workload, which diminishes the usability of the overall model building process. In this paper, we develop LookHere, which integrates in-situ object annotations into vision-based IMT. LookHere exploits users' deictic gestures to segment the objects of interest in real time. This segmentation information can be additionally used for training. To achieve the reliable performance of this object segmentation, we utilize our custom dataset called HuTics, including 2040 front-facing images of deictic gestures toward various objects by 170 people. The quantitative results of our user study showed that participants were 16.3 times faster in creating a model with our system compared to a standard IMT system with a post-hoc annotation process while demonstrating comparable accuracies. Additionally, models created by our system showed a significant accuracy improvement (Delta mIoU=0.466) in segmenting the objects of interest compared to those without annotations.