# RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching

arXiv: https://arxiv.org/abs/2008.08688 · HTML: https://mlsanity.com/paper/2008.08688/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Aug 2020.

## Built on by (most cited first, 14 of 14)

- [Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces](https://mlsanity.com/paper/2203.03254/) (2022)
- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling](https://mlsanity.com/paper/2208.06350/) (2022)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams](https://mlsanity.com/paper/2405.18614/) (2024)
- [DrawTalking: Building Interactive Worlds by Sketching and Speaking](https://mlsanity.com/paper/2401.05631/) (2024)
- [Augmented Conversation with Embedded Speech-Driven On-the-Fly Referencing in AR](https://mlsanity.com/paper/2405.18537/) (2024)
- [Multimodal 3D Fusion and In-Situ Learning for Spatially Aware AI](https://mlsanity.com/paper/2410.04652/) (2024)
- [Multi-Object Sketch Animation by Scene Decomposition and Motion Planning](https://mlsanity.com/paper/2503.19351/) (2025)
- [AI-Gadget Kit: Integrating Swarm User Interfaces with LLM-driven Agents for Rich Tabletop Game Applications](https://mlsanity.com/paper/2407.17086/) (2024)
- [Empowering Children to Create AI-Enabled Augmented Reality Experiences](https://mlsanity.com/paper/2508.08467/) (2025)
- [SketchPlay: Intuitive Creation of Physically Realistic VR Content with Gesture-Driven Sketching](https://mlsanity.com/paper/2512.22016/) (2025)
- [Spatula: Exploring On-Demand In-Situ Interfaces and Interaction for Attribute Control](https://mlsanity.com/paper/2607.10405/) (2026)

## Abstract (arXiv)

We present RealitySketch, an augmented reality interface for sketching interactive graphics and visualizations. In recent years, an increasing number of AR sketching tools enable users to draw and embed sketches in the real world. However, with the current tools, sketched contents are inherently static, floating in mid air without responding to the real world. This paper introduces a new way to embed dynamic and responsive graphics in the real world. In RealitySketch, the user draws graphical elements on a mobile AR screen and binds them with physical objects in real-time and improvisational ways, so that the sketched elements dynamically move with the corresponding physical motion. The user can also quickly visualize and analyze real-world phenomena through responsive graph plots or interactive visualizations. This paper contributes to a set of interaction techniques that enable capturing, parameterizing, and visualizing real-world motion without pre-defined programs and configurations. Finally, we demonstrate our tool with several application scenarios, including physics education, sports training, and in-situ tangible interfaces.