# Enhancing Sign Language Teaching: A Mixed Reality Approach for Immersive Learning and Multi-Dimensional Feedback

arXiv: https://arxiv.org/abs/2404.10490 · HTML: https://mlsanity.com/paper/2404.10490/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 23 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Builds on (most cited first, 8 of 8)

- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Learning to Estimate 3D Hand Pose from Single RGB Images](https://mlsanity.com/paper/1705.01389/) (2017)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [One-Stage 3D Whole-Body Mesh Recovery with Component Aware Transformer](https://mlsanity.com/paper/2303.16160/) (2023)
- [Learning Regularity in Skeleton Trajectories for Anomaly Detection in Videos](https://mlsanity.com/paper/1903.03295/) (2019)
- [Video-based Sign Language Recognition without Temporal Segmentation](https://mlsanity.com/paper/1801.10111/) (2018)
- [SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos](https://mlsanity.com/paper/2112.13715/) (2021)
- [Using a single RGB frame for real time 3D hand pose estimation in the wild](https://mlsanity.com/paper/1712.03866/) (2017)

## Built on by (most cited first, 1 of 1)

- [Behavioral Engagement in VR-Based Sign Language Learning: Visual Attention as a Predictor of Performance and Temporal Dynamics](https://mlsanity.com/paper/2603.19535/) (2026)

## Abstract (arXiv)

Traditional sign language teaching methods face challenges such as limited feedback and diverse learning scenarios. Although 2D resources lack real-time feedback, classroom teaching is constrained by a scarcity of teacher. Methods based on VR and AR have relatively primitive interaction feedback mechanisms. This study proposes an innovative teaching model that uses real-time monocular vision and mixed reality technology. First, we introduce an improved hand-posture reconstruction method to achieve sign language semantic retention and real-time feedback. Second, a ternary system evaluation algorithm is proposed for a comprehensive assessment, maintaining good consistency with experts in sign language. Furthermore, we use mixed reality technology to construct a scenario-based 3D sign language classroom and explore the user experience of scenario teaching. Overall, this paper presents a novel teaching method that provides an immersive learning experience, advanced posture reconstruction, and precise feedback, achieving positive feedback on user experience and learning effectiveness.