# AUGlasses: Continuous Action Unit based Facial Reconstruction with Low-power IMUs on Smart Glasses

arXiv: https://arxiv.org/abs/2405.13289 · HTML: https://mlsanity.com/paper/2405.13289/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 69 references that are in the map.
- First posted on arXiv 22 May 2024.

## Builds on (most cited first, 14 of 14)

- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Learning Detailed Face Reconstruction from a Single Image](https://mlsanity.com/paper/1611.05053/) (2016)
- [Style Aggregated Network for Facial Landmark Detection](https://mlsanity.com/paper/1803.04108/) (2018)
- [Machine Translation Decoding beyond Beam Search](https://mlsanity.com/paper/2104.05336/) (2021)
- [What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks?](https://mlsanity.com/paper/2202.05998/) (2022)
- [DAD-3DHeads: A Large-scale Dense, Accurate and Diverse Dataset for 3D Head Alignment from a Single Image](https://mlsanity.com/paper/2204.03688/) (2022)
- [Understanding and Improving Recurrent Networks for Human Activity Recognition by Continuous Attention](https://mlsanity.com/paper/1810.04038/) (2018)
- [Transformer-Based Self-Supervised Learning for Emotion Recognition](https://mlsanity.com/paper/2204.05103/) (2022)
- [Confidence-Aware Scheduled Sampling for Neural Machine Translation](https://mlsanity.com/paper/2107.10427/) (2021)
- [Face Alignment using a 3D Deeply-initialized Ensemble of Regression Trees](https://mlsanity.com/paper/1902.01831/) (2019)
- [InMyFace: Inertial and Mechanomyography-Based Sensor Fusion for Wearable Facial Activity Recognition](https://mlsanity.com/paper/2302.04024/) (2023)
- [WiFiMod: Transformer-based Indoor Human Mobility Modeling using Passive Sensing](https://mlsanity.com/paper/2104.09835/) (2021)
- [Fiducial Focus Augmentation for Facial Landmark Detection](https://mlsanity.com/paper/2402.15044/) (2024)
- [Dense 3D Facial Reconstruction from a Single Depth Image in Unconstrained Environment](https://mlsanity.com/paper/1704.07142/) (2017)

## Abstract (arXiv)

Recent advancements in augmented reality (AR) have enabled the use of various sensors on smart glasses for applications like facial reconstruction, which is vital to improve AR experiences for virtual social activities. However, the size and power constraints of smart glasses demand a miniature and low-power sensing solution. AUGlasses achieves unobtrusive low-power facial reconstruction by placing inertial measurement units (IMU) against the temporal area on the face to capture the skin deformations, which are caused by facial muscle movements. These IMU signals, along with historical data on facial action units (AUs), are processed by a transformer-based deep learning model to estimate AU intensities in real-time, which are then used for facial reconstruction. Our results show that AUGlasses accurately predicts the strength (0-5 scale) of 14 key AUs with a cross-user mean absolute error (MAE) of 0.187 (STD = 0.025) and achieves facial reconstruction with a cross-user MAE of 1.93 mm (STD = 0.353). We also integrated various preprocessing and training techniques to ensure robust performance for continuous sensing. Micro-benchmark tests indicate that our system consistently performs accurate continuous facial reconstruction with a fine-tuned cross-user model, achieving an AU MAE of 0.35.