# LipLearner: Customizable Silent Speech Interactions on Mobile Devices

arXiv: https://arxiv.org/abs/2302.05907 · HTML: https://mlsanity.com/paper/2302.05907/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 67 references that are in the map.
- First posted on arXiv 12 Feb 2023.

## Builds on (most cited first, 19 of 19)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [A Closer Look at Few-shot Classification](https://mlsanity.com/paper/1904.04232/) (2019)
- [MediaPipe: A Framework for Building Perception Pipelines](https://mlsanity.com/paper/1906.08172/) (2019)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency](https://mlsanity.com/paper/2101.03149/) (2021)
- [Lipreading using Temporal Convolutional Networks](https://mlsanity.com/paper/2001.08702/) (2020)
- [Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis](https://mlsanity.com/paper/2005.08209/) (2020)
- [End-to-end Audiovisual Speech Recognition](https://mlsanity.com/paper/1802.06424/) (2018)
- [Contrastive Predictive Coding for Human Activity Recognition](https://mlsanity.com/paper/2012.05333/) (2020)
- [Learn an Effective Lip Reading Model without Pains](https://mlsanity.com/paper/2011.07557/) (2020)
- [Enabling hand gesture customization on wrist-worn devices](https://mlsanity.com/paper/2203.15239/) (2022)
- [ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Users](https://mlsanity.com/paper/2202.11134/) (2022)
- [Updating the silent speech challenge benchmark with deep learning](https://mlsanity.com/paper/1709.06818/) (2017)
- [VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversion](https://mlsanity.com/paper/2202.09081/) (2022)

## Built on by (most cited first, 5 of 5)

- [LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognition](https://mlsanity.com/paper/2501.04204/) (2025)
- [HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Reality](https://mlsanity.com/paper/2410.16430/) (2024)
- [RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representations](https://mlsanity.com/paper/2307.01233/) (2023)
- [Single-tap Latency Reduction with Single- or Double- tap Prediction](https://mlsanity.com/paper/2408.02525/) (2024)
- [NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interaction](https://mlsanity.com/paper/2603.10324/) (2026)

## Abstract (arXiv)

Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.