# SpeakingFaces: A Large-Scale Multimodal Dataset of Voice Commands with Visual and Thermal Video Streams

arXiv: https://arxiv.org/abs/2012.02961 · HTML: https://mlsanity.com/paper/2012.02961/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Dec 2020.

## Built on by (most cited first, 21 of 21)

- [Synthetic Thermal and RGB Videos for Automatic Pain Assessment utilizing a Vision-MLP Architecture](https://mlsanity.com/paper/2407.19811/) (2024)
- [MAGID: An Automated Pipeline for Generating Synthetic Multi-modal Datasets](https://mlsanity.com/paper/2403.03194/) (2024)
- [PainFormer: a Vision Foundation Model for Automatic Pain Assessment](https://mlsanity.com/paper/2505.01571/) (2025)
- [Beyond the Visible: A Survey on Cross-spectral Face Recognition](https://mlsanity.com/paper/2201.04435/) (2022)
- [A Study of Multilingual End-to-End Speech Recognition for Kazakh, Russian, and English](https://mlsanity.com/paper/2108.01280/) (2021)
- [Self-adversarial Multi-scale Contrastive Learning for Semantic Segmentation of Thermal Facial Images](https://mlsanity.com/paper/2209.10700/) (2022)
- [A Multimodal Sensor Fusion Framework Robust to Missing Modalities for Person Recognition](https://mlsanity.com/paper/2210.10972/) (2022)
- [A Survey on Computer Vision based Human Analysis in the COVID-19 Era](https://mlsanity.com/paper/2211.03705/) (2022)
- [Modularity in Deep Learning: A Survey](https://mlsanity.com/paper/2310.01154/) (2023)
- [Learning to Find Missing Video Frames with Synthetic Data Augmentation: A General Framework and Application in Generating Thermal Images Using RGB Cameras](https://mlsanity.com/paper/2403.00196/) (2024)
- [Thermal Imaging for Contactless Cardiorespiratory and Sudomotor Response Monitoring](https://mlsanity.com/paper/2602.12361/) (2026)
- [MTVDiff: Multimodal Conditional Latent Diffusion for Enhanced Thermal-to-Visible Face Translation](https://mlsanity.com/paper/2607.19886/) (2026)
- [A Study of Multimodal Person Verification Using Audio-Visual-Thermal Data](https://mlsanity.com/paper/2110.12136/) (2021)
- [Biometrics in the Time of Pandemic: 40% Masked Face Recognition Degradation can be Reduced to 2%](https://mlsanity.com/paper/2201.00461/) (2022)
- [A Novel Fully Annotated Thermal Infrared Face Dataset: Recorded in Various Environment Conditions and Distances From The Camera](https://mlsanity.com/paper/2205.02093/) (2022)
- [Fairness on Synthetic Visual and Thermal Mask Images](https://mlsanity.com/paper/2209.08762/) (2022)
- [MMSD-Net: Towards Multi-modal Stuttering Detection](https://mlsanity.com/paper/2407.11492/) (2024)
- [LaPIG: Cross-Modal Generation of Paired Thermal and Visible Facial Images](https://mlsanity.com/paper/2503.16376/) (2025)
- [Multi-Attribute guided Thermal Face Image Translation based on Latent Diffusion Model](https://mlsanity.com/paper/2512.21032/) (2025)
- [ThermEval: A Structured Benchmark for Evaluation of Vision-Language Models on Thermal Imagery](https://mlsanity.com/paper/2602.14989/) (2026)
- [A Comprehensive Review of Multimodal Facial State Analysis: Tasks, Methods, and Resources](https://mlsanity.com/paper/2609.13255/) (2026)

## Abstract (arXiv)

We present SpeakingFaces as a publicly-available large-scale multimodal dataset developed to support machine learning research in contexts that utilize a combination of thermal, visual, and audio data streams; examples include human-computer interaction, biometric authentication, recognition systems, domain transfer, and speech recognition. SpeakingFaces is comprised of aligned high-resolution thermal and visual spectra image streams of fully-framed faces synchronized with audio recordings of each subject speaking approximately 100 imperative phrases. Data were collected from 142 subjects, yielding over 13,000 instances of synchronized data (~3.8 TB). For technical validation, we demonstrate two baseline examples. The first baseline shows classification by gender, utilizing different combinations of the three data streams in both clean and noisy environments. The second example consists of thermal-to-visual facial image translation, as an instance of domain transfer.