# Augmented Physics: Creating Interactive and Embedded Physics Simulations from Static Textbook Diagrams

arXiv: https://arxiv.org/abs/2405.18614 · HTML: https://mlsanity.com/paper/2405.18614/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 72 references that are in the map.
- First posted on arXiv 28 May 2024.

## Builds on (most cited first, 9 of 9)

- [Scim: Intelligent Skimming Support for Scientific Papers](https://mlsanity.com/paper/2205.04561/) (2022)
- [Threddy: An Interactive System for Personalized Thread-based Exploration and Organization of Scientific Literature](https://mlsanity.com/paper/2208.03455/) (2022)
- [RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching](https://mlsanity.com/paper/2008.08688/) (2020)
- [Kori: Interactive Synthesis of Text and Charts in Data Documents](https://mlsanity.com/paper/2108.04203/) (2021)
- [Augmented Math: Authoring AR-Based Explorable Explanations by Augmenting Static Math Textbooks](https://mlsanity.com/paper/2307.16112/) (2023)
- [CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration](https://mlsanity.com/paper/2308.03311/) (2023)
- [Sketched Reality: Sketching Bi-Directional Interactions Between Virtual and Physical Worlds with AR and Actuated Tangible UI](https://mlsanity.com/paper/2208.06341/) (2022)
- [Chalktalk : A Visualization and Communication Language -- As a Tool in the Domain of Computer Science Education](https://mlsanity.com/paper/1809.07166/) (2018)
- [RealityCanvas: Augmented Reality Sketching for Embedded and Responsive Scribble Animation Effects](https://mlsanity.com/paper/2307.16116/) (2023)

## Built on by (most cited first, 6 of 6)

- [MapStory: Prototyping Editable Map Animations with LLM Agents](https://mlsanity.com/paper/2505.21966/) (2025)
- [Empowering Children to Create AI-Enabled Augmented Reality Experiences](https://mlsanity.com/paper/2508.08467/) (2025)
- [Explorable Proofs: Making Written Proofs Explorable by Grounding Them in Formal Representations](https://mlsanity.com/paper/2604.02598/) (2026)
- [Spatula: Exploring On-Demand In-Situ Interfaces and Interaction for Attribute Control](https://mlsanity.com/paper/2607.10405/) (2026)
- [CircuTutor: Transforming Static Circuit Problems into Intelligent and Dynamic Tutoring](https://mlsanity.com/paper/2609.08254/) (2026)
- [Generative Tutorial: Towards Live Contextualized Visual Instructions for Physical Tasks](https://mlsanity.com/paper/2609.24955/) (2026)

## Abstract (arXiv)

We introduce Augmented Physics, a machine learning-integrated authoring tool designed for creating embedded interactive physics simulations from static textbook diagrams. Leveraging recent advancements in computer vision, such as Segment Anything and Multi-modal LLMs, our web-based system enables users to semi-automatically extract diagrams from physics textbooks and generate interactive simulations based on the extracted content. These interactive diagrams are seamlessly integrated into scanned textbook pages, facilitating interactive and personalized learning experiences across various physics concepts, such as optics, circuits, and kinematics. Drawing from an elicitation study with seven physics instructors, we explore four key augmentation strategies: 1) augmented experiments, 2) animated diagrams, 3) bi-directional binding, and 4) parameter visualization. We evaluate our system through technical evaluation, a usability study (N=12), and expert interviews (N=12). Study findings suggest that our system can facilitate more engaging and personalized learning experiences in physics education.