# Smile Like You Mean It: Driving Animatronic Robotic Face with Learned Models

arXiv: https://arxiv.org/abs/2105.12724 · HTML: https://mlsanity.com/paper/2105.12724/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 74 references that are in the map.
- First posted on arXiv 26 May 2021.

## Builds on (most cited first, 21 of 21)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [FlowNet: Learning Optical Flow with Convolutional Networks](https://mlsanity.com/paper/1504.06852/) (2015)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [Face2Face: Real-time Face Capture and Reenactment of RGB Videos](https://mlsanity.com/paper/2007.14808/) (2020)
- [End-to-end Driving via Conditional Imitation Learning](https://mlsanity.com/paper/1710.02410/) (2017)
- [An Algorithmic Perspective on Imitation Learning](https://mlsanity.com/paper/1811.06711/) (2018)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution](https://mlsanity.com/paper/1807.03247/) (2018)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Hand Keypoint Detection in Single Images using Multiview Bootstrapping](https://mlsanity.com/paper/1704.07809/) (2017)
- [Animating Arbitrary Objects via Deep Motion Transfer](https://mlsanity.com/paper/1812.08861/) (2018)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Dance Dance Generation: Motion Transfer for Internet Videos](https://mlsanity.com/paper/1904.00129/) (2019)
- [Vid2Game: Controllable Characters Extracted from Real-World Videos](https://mlsanity.com/paper/1904.08379/) (2019)
- [Agile Autonomous Driving using End-to-End Deep Imitation Learning](https://mlsanity.com/paper/1709.07174/) (2017)

## Built on by (most cited first, 12 of 12)

- [Full-Body Visual Self-Modeling of Robot Morphologies](https://mlsanity.com/paper/2111.06389/) (2021)
- [Optimizing Facial Expressions of an Android Robot Effectively: a Bayesian Optimization Approach](https://mlsanity.com/paper/2301.05620/) (2023)
- [Driving Animatronic Robot Facial Expression From Speech](https://mlsanity.com/paper/2403.12670/) (2024)
- [HAPI: A Model for Learning Robot Facial Expressions from Human Preferences](https://mlsanity.com/paper/2503.17046/) (2025)
- [ExFace: Expressive Facial Control for Humanoid Robots with Diffusion Transformers and Bootstrap Training](https://mlsanity.com/paper/2504.14477/) (2025)
- [X2C: A Large-Scale Benchmark for Nuanced Humanoid Facial Expression Imitation](https://mlsanity.com/paper/2505.11146/) (2025)
- [Knolling Bot: Teaching Robots the Human Notion of Tidiness](https://mlsanity.com/paper/2310.04566/) (2023)
- [Learning to Control an Android Robot Head for Facial Animation](https://mlsanity.com/paper/2412.13641/) (2024)
- [Learning High-Fidelity Robot Self-Model with Articulated 3D Gaussian Splatting](https://mlsanity.com/paper/2503.05398/) (2025)
- [SingingBot: An Avatar-Driven System for Robotic Face Singing Performance](https://mlsanity.com/paper/2601.02125/) (2026)
- [VividFace: Real-Time and Realistic Facial Expression Shadowing for Humanoid Robots](https://mlsanity.com/paper/2602.07506/) (2026)
- [Eye, Robot: Learning Expressive Gaze Dynamics via a VLM-Guided Transformer-VAE](https://mlsanity.com/paper/2610.19635/) (2026)

## Abstract (arXiv)

Ability to generate intelligent and generalizable facial expressions is essential for building human-like social robots. At present, progress in this field is hindered by the fact that each facial expression needs to be programmed by humans. In order to adapt robot behavior in real time to different situations that arise when interacting with human subjects, robots need to be able to train themselves without requiring human labels, as well as make fast action decisions and generalize the acquired knowledge to diverse and new contexts. We addressed this challenge by designing a physical animatronic robotic face with soft skin and by developing a vision-based self-supervised learning framework for facial mimicry. Our algorithm does not require any knowledge of the robot's kinematic model, camera calibration or predefined expression set. By decomposing the learning process into a generative model and an inverse model, our framework can be trained using a single motor babbling dataset. Comprehensive evaluations show that our method enables accurate and diverse face mimicry across diverse human subjects. The project website is at http://www.cs.columbia.edu/~bchen/aiface/