# Integrating Persian Lip Reading in Surena-V Humanoid Robot for Human-Robot Interaction

arXiv: https://arxiv.org/abs/2501.13996 · HTML: https://mlsanity.com/paper/2501.13996/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 23 Jan 2025.

## Builds on (most cited first, 5 of 5)

- [Lip Reading Using Convolutional Auto Encoders as Feature Extractor](https://mlsanity.com/paper/1805.12371/) (2018)
- [Advances and Challenges in Deep Lip Reading](https://mlsanity.com/paper/2110.07879/) (2021)
- [Deep Learning for Lip Reading using Audio-Visual Information for Urdu Language](https://mlsanity.com/paper/1802.05521/) (2018)
- [LipReading with 3D-2D-CNN BLSTM-HMM and word-CTC models](https://mlsanity.com/paper/1906.12170/) (2019)
- [Online Bipedal Locomotion Adaptation for Stepping on Obstacles Using a Novel Foot Sensor](https://mlsanity.com/paper/2212.13416/) (2022)

## Abstract (arXiv)

Lip reading is vital for robots in social settings, improving their ability to understand human communication. This skill allows them to communicate more easily in crowded environments, especially in caregiving and customer service roles. Generating a Persian Lip-reading dataset, this study integrates Persian lip-reading technology into the Surena-V humanoid robot to improve its speech recognition capabilities. Two complementary methods are explored, an indirect method using facial landmark tracking and a direct method leveraging convolutional neural networks (CNNs) and long short-term memory (LSTM) networks. The indirect method focuses on tracking key facial landmarks, especially around the lips, to infer movements, while the direct method processes raw video data for action and speech recognition. The best-performing model, LSTM, achieved 89% accuracy and has been successfully implemented into the Surena-V robot for real-time human-robot interaction. The study highlights the effectiveness of these methods, particularly in environments where verbal communication is limited.