# Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks

arXiv: https://arxiv.org/abs/2307.16112 · HTML: https://mlsanity.com/paper/2307.16112/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 68 references that are in the map.
- First posted on arXiv 30 Jul 2023.

## Builds on (most cited first, 9 of 9)

- [CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualization](https://mlsanity.com/paper/2004.15004/) (2020)
- [Direct-Manipulation Visualization of Deep Networks](https://mlsanity.com/paper/1708.03788/) (2017)
- [RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching](https://mlsanity.com/paper/2008.08688/) (2020)
- [RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling](https://mlsanity.com/paper/2208.06350/) (2022)
- [Teachable Reality: Prototyping Tangible Augmented Reality with Everyday Objects by Leveraging Interactive Machine Teaching](https://mlsanity.com/paper/2302.11046/) (2023)
- [Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI](https://mlsanity.com/paper/2208.06341/) (2022)
- [TraceDiff: Debugging Unexpected Code Behavior Using Trace Divergences](https://mlsanity.com/paper/1708.03786/) (2017)
- [Fidyll: A Compiler for Cross-Format Data Stories & Explorable Explanations](https://mlsanity.com/paper/2205.09858/) (2022)
- [Learnable Programming: Blocks and Beyond](https://mlsanity.com/paper/1705.09413/) (2017)

## Built on by (most cited first, 7 of 7)

- [Augmented Object Intelligence with XR-Objects](https://mlsanity.com/paper/2404.13274/) (2024)
- [Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams](https://mlsanity.com/paper/2405.18614/) (2024)
- [Everyday AR through AI-in-the-Loop](https://mlsanity.com/paper/2412.12681/) (2024)
- [Reality Proxy: Fluid Interactions with Real-World Objects in MR via Abstract Representations](https://mlsanity.com/paper/2507.17248/) (2025)
- [RealitySummary: Exploring On-Demand Mixed Reality Text Summarization and Question Answering using Large Language Models](https://mlsanity.com/paper/2405.18620/) (2024)
- [Empowering Children to Create AI-Enabled Augmented Reality Experiences](https://mlsanity.com/paper/2508.08467/) (2025)
- [CircuTutor: Transforming Static Circuit Problems into Intelligent and Dynamic Tutoring](https://mlsanity.com/paper/2609.08254/) (2026)

## Abstract (arXiv)

We introduce Augmented Math, a machine learning-based approach to authoring AR explorable explanations by augmenting static math textbooks without programming. To augment a static document, our system first extracts mathematical formulas and figures from a given document using optical character recognition (OCR) and computer vision. By binding and manipulating these extracted contents, the user can see the interactive animation overlaid onto the document through mobile AR interfaces. This empowers non-technical users, such as teachers or students, to transform existing math textbooks and handouts into on-demand and personalized explorable explanations. To design our system, we first analyzed existing explorable math explanations to identify common design strategies. Based on the findings, we developed a set of augmentation techniques that can be automatically generated based on the extracted content, which are 1) dynamic values, 2) interactive figures, 3) relationship highlights, 4) concrete examples, and 5) step-by-step hints. To evaluate our system, we conduct two user studies: preliminary user testing and expert interviews. The study results confirm that our system allows more engaging experiences for learning math concepts.