# A First Step in Using Machine Learning Methods to Enhance Interaction Analysis for Embodied Learning Environments

arXiv: https://arxiv.org/abs/2405.06203 · HTML: https://mlsanity.com/paper/2405.06203/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 27 references that are in the map.
- First posted on arXiv 10 May 2024.

## Builds on (most cited first, 4 of 4)

- [Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networks](https://mlsanity.com/paper/1604.02878/) (2016)
- [ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth](https://mlsanity.com/paper/2302.12288/) (2023)
- [Gaze360: Physically Unconstrained Gaze Estimation in the Wild](https://mlsanity.com/paper/1910.10088/) (2019)
- [L2CS-Net: Fine-Grained Gaze Estimation in Unconstrained Environments](https://mlsanity.com/paper/2203.03339/) (2022)

## Built on by (most cited first, 6 of 6)

- [Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review](https://mlsanity.com/paper/2408.14491/) (2024)
- [LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports](https://mlsanity.com/paper/2503.02099/) (2025)
- [3D Gaze Tracking for Studying Collaborative Interactions in Mixed-Reality Environments](https://mlsanity.com/paper/2406.11003/) (2024)
- [Vision6D: 3D-to-2D Interactive Visualization and Annotation Tool for 6D Pose Estimation](https://mlsanity.com/paper/2504.15329/) (2025)
- [Using Large Language Models to Detect Socially Shared Regulation of Collaborative Learning](https://mlsanity.com/paper/2601.04458/) (2026)
- [AI Misuse in Education Is a Measurement Problem: Toward a Learning Visibility Framework](https://mlsanity.com/paper/2603.07834/) (2026)

## Abstract (arXiv)

Investigating children's embodied learning in mixed-reality environments, where they collaboratively simulate scientific processes, requires analyzing complex multimodal data to interpret their learning and coordination behaviors. Learning scientists have developed Interaction Analysis (IA) methodologies for analyzing such data, but this requires researchers to watch hours of videos to extract and interpret students' learning patterns. Our study aims to simplify researchers' tasks, using Machine Learning and Multimodal Learning Analytics to support the IA processes. Our study combines machine learning algorithms and multimodal analyses to support and streamline researcher efforts in developing a comprehensive understanding of students' scientific engagement through their movements, gaze, and affective responses in a simulated scenario. To facilitate an effective researcher-AI partnership, we present an initial case study to determine the feasibility of visually representing students' states, actions, gaze, affect, and movement on a timeline. Our case study focuses on a specific science scenario where students learn about photosynthesis. The timeline allows us to investigate the alignment of critical learning moments identified by multimodal and interaction analysis, and uncover insights into students' temporal learning progressions.