# Enabling Automatic Self-Talk Detection via Earables

arXiv: https://arxiv.org/abs/2511.07493 · HTML: https://mlsanity.com/paper/2511.07493/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 72 references that are in the map.
- First posted on arXiv 10 Nov 2025.

## Builds on (most cited first, 12 of 12)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Emotion-LLaMA: Multimodal Emotion Recognition and Reasoning with Instruction Tuning](https://mlsanity.com/paper/2406.11161/) (2024)
- [Towards end-to-end spoken language understanding](https://mlsanity.com/paper/1802.08395/) (2018)
- [A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding](https://mlsanity.com/paper/2111.02735/) (2021)
- [GFF: Gated Fully Fusion for Semantic Segmentation](https://mlsanity.com/paper/1904.01803/) (2019)
- [End-to-End Neural Transformer Based Spoken Language Understanding](https://mlsanity.com/paper/2008.10984/) (2020)
- [Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems](https://mlsanity.com/paper/2112.05842/) (2021)
- [DeepMood: Modeling Mobile Phone Typing Dynamics for Mood Detection](https://mlsanity.com/paper/1803.08986/) (2018)
- [Improving Irregularly Sampled Time Series Learning with Dense Descriptors of Time](https://mlsanity.com/paper/2003.09291/) (2020)
- [Speech Understanding on Tiny Devices with A Learning Cache](https://mlsanity.com/paper/2311.18188/) (2023)

## Abstract (arXiv)

Self-talk-an internal dialogue that can occur silently or be spoken aloud-plays a crucial role in emotional regulation, cognitive processing, and motivation, yet has remained largely invisible and unmeasurable in everyday life. In this paper, we present MutterMeter, a mobile system that automatically detects vocalized self-talk from audio captured by earable microphones in real-world settings. Detecting self-talk is technically challenging due to its diverse acoustic forms, semantic and grammatical incompleteness, and irregular occurrence patterns, which differ fundamentally from assumptions underlying conventional speech understanding models. To address these challenges, MutterMeter employs a hierarchical classification architecture that progressively integrates acoustic, linguistic, and contextual information through a sequential processing pipeline, adaptively balancing accuracy and computational efficiency. We build and evaluate MutterMeter using a first-of-its-kind dataset comprising 31.1 hours of audio collected from 25 participants. Experimental results demonstrate that MutterMeter achieves robust performance with a macro-averaged F1 score of 0.84, outperforming conventional approaches, including LLM-based and speech emotion recognition models.