# Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching

arXiv: https://arxiv.org/abs/2302.11046 · HTML: https://mlsanity.com/paper/2302.11046/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 87 references that are in the map.
- First posted on arXiv 21 Feb 2023.

## Builds on (most cited first, 9 of 9)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Fast R-CNN](https://mlsanity.com/paper/1504.08083/) (2015)
- [RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching](https://mlsanity.com/paper/2008.08688/) (2020)
- [RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling](https://mlsanity.com/paper/2208.06350/) (2022)
- [InfraredTags: Embedding Invisible AR Markers and Barcodes Using Low-Cost, Infrared-Based 3D Printing and Imaging Tools](https://mlsanity.com/paper/2202.06165/) (2022)
- [Gesture-aware Interactive Machine Teaching with In-situ Object Annotations](https://mlsanity.com/paper/2208.01211/) (2022)
- [Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI](https://mlsanity.com/paper/2208.06341/) (2022)

## Built on by (most cited first, 7 of 7)

- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [Augmented Conversation with Embedded Speech-Driven On-the-Fly Referencing in AR](https://mlsanity.com/paper/2405.18537/) (2024)
- [Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations](https://mlsanity.com/paper/2507.17248/) (2025)
- [RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models](https://mlsanity.com/paper/2405.18620/) (2024)
- [Augmenting Human Cognition through Everyday AR](https://mlsanity.com/paper/2505.03492/) (2025)
- [Empowering Children to Create AI-Enabled Augmented Reality Experiences](https://mlsanity.com/paper/2508.08467/) (2025)

## Abstract (arXiv)

This paper introduces Teachable Reality, an augmented reality (AR) prototyping tool for creating interactive tangible AR applications with arbitrary everyday objects. Teachable Reality leverages vision-based interactive machine teaching (e.g., Teachable Machine), which captures real-world interactions for AR prototyping. It identifies the user-defined tangible and gestural interactions using an on-demand computer vision model. Based on this, the user can easily create functional AR prototypes without programming, enabled by a trigger-action authoring interface. Therefore, our approach allows the flexibility, customizability, and generalizability of tangible AR applications that can address the limitation of current marker-based approaches. We explore the design space and demonstrate various AR prototypes, which include tangible and deformable interfaces, context-aware assistants, and body-driven AR applications. The results of our user study and expert interviews confirm that our approach can lower the barrier to creating functional AR prototypes while also allowing flexible and general-purpose prototyping experiences.