# Bridging the Communication Gap: Artificial Agents Learning Sign Language through Imitation

arXiv: https://arxiv.org/abs/2406.10043 · HTML: https://mlsanity.com/paper/2406.10043/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 11 of its 38 references that are in the map.
- First posted on arXiv 14 Jun 2024.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 11 of 11)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [End-to-end Recovery of Human Shape and Pose](https://mlsanity.com/paper/1712.06584/) (2017)
- [FrankMocap: A Monocular 3D Whole-Body Pose Estimation System via Regression and Integration](https://mlsanity.com/paper/2108.06428/) (2021)
- [A Review of Verbal and Non-Verbal Human-Robot Interactive Communication](https://mlsanity.com/paper/1401.4994/) (2014)
- [Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language](https://mlsanity.com/paper/2103.08882/) (2021)
- [Dynamic Movement Primitive based Motion Retargeting for Dual-Arm Sign Language Motions](https://mlsanity.com/paper/2011.03914/) (2020)
- [WLASL-LEX: a Dataset for Recognising Phonological Properties in American Sign Language](https://mlsanity.com/paper/2203.06096/) (2022)
- [Phonology Recognition in American Sign Language](https://mlsanity.com/paper/2110.00453/) (2021)
- [Quick, Stat!: A Statistical Analysis of the Quick, Draw! Dataset](https://mlsanity.com/paper/1907.06417/) (2019)
- [Sign Language Representation by TEO Humanoid Robot: End-User Interest, Comprehension and Satisfaction](https://mlsanity.com/paper/1901.05939/) (2019)

## Abstract (arXiv)

Artificial agents, particularly humanoid robots, interact with their environment, objects, and people using cameras, actuators, and physical presence. Their communication methods are often pre-programmed, limiting their actions and interactions. Our research explores acquiring non-verbal communication skills through learning from demonstrations, with potential applications in sign language comprehension and expression. In particular, we focus on imitation learning for artificial agents, exemplified by teaching a simulated humanoid American Sign Language. We use computer vision and deep learning to extract information from videos, and reinforcement learning to enable the agent to replicate observed actions. Compared to other methods, our approach eliminates the need for additional hardware to acquire information. We demonstrate how the combination of these different techniques offers a viable way to learn sign language. Our methodology successfully teaches 5 different signs involving the upper body (i.e., arms and hands). This research paves the way for advanced communication skills in artificial agents.