mlsanityData to 14 Oct 2026

LipLearner: Customizable Silent Speech Interactions on Mobile Devices

arXiv 2302.05907 ↗First posted 12 Feb 2023cs.HC cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Silent speech interface is a promising technology that enables private communications in natural language. However, previous approaches only support a small and inflexible vocabulary, which leads to limited expressiveness. We leverage contrastive learning to learn efficient lipreading representations, enabling few-shot command customization with minimal user effort. Our model exhibits high robustness to different lighting, posture, and gesture conditions on an in-the-wild dataset. For 25-command classification, an F1-score of 0.8947 is achievable only using one shot, and its performance can be further boosted by adaptively learning from more data. This generalizability allowed us to develop a mobile silent speech interface empowered with on-device fine-tuning and visual keyword spotting. A user study demonstrated that with LipLearner, users could define their own commands with high reliability guaranteed by an online incremental learning scheme. Subjective feedback indicated that our system provides essential functionalities for customizable silent speech interactions with high usability and learnability.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 19 of its 67 references that are in the map. First posted on arXiv 12 Feb 2023.

Builds on

19 of 67 references.

References · 19

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  6. 2019A Closer Look at Few-shot Classificationcited by 684
  7. 2019MediaPipe: A Framework for Building Perception Pipelinescited by 583
  8. 2017Objects that Soundcited by 256
  9. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  10. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  11. 2020Lipreading using Temporal Convolutional Networkscited by 102
  12. 2020Learning Individual Speaking Styles for Accurate Lip to Speech Synthesiscited by 74
7 more
  1. 2018End-to-end Audiovisual Speech Recognitioncited by 73
  2. 2020Contrastive Predictive Coding for Human Activity Recognitioncited by 55
  3. 2020Learn an Effective Lip Reading Model without Painscited by 19
  4. 2022Enabling hand gesture customization on wrist-worn devicescited by 15
  5. 2022ProtoSound: A Personalized and Scalable Sound Recognition System for Deaf and Hard-of-Hearing Userscited by 8
  6. 2017Updating the silent speech challenge benchmark with deep learningcited by 7
  7. 2022VCVTS: Multi-speaker Video-to-Speech synthesis via cross-modal knowledge transfer from voice conversioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 2; 2025: 1; 2026: 1.

5 papers.

  1. 2025LipGen: Viseme-Guided Lip Video Generation for Enhancing Visual Speech Recognitioncited by 4
  2. 2024HaHeAE: Learning Generalisable Joint Representations of Human Hand and Head Movements in Extended Realitycited by 2
  3. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  4. 2024Single-tap Latency Reduction with Single- or Double- tap Predictioncited by 0
  5. 2026NasoVoce: A Nose-Mounted Low-Audibility Speech Interface for Always-Available Speech Interactioncited by 0