mlsanityData to 7 Oct 2026

Robust Multi-Tier Infant-Centered Audio Understanding with Whisper via Structured Speaker Conditioning

arXiv 2608.11587 ↗First posted 12 Aug 2026eess.AS cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in model design and self-supervised audio representations have improved speech and audio understanding, yet infant-centered naturalistic recordings remain challenging due to limited labeled data, low signal-to-noise ratio, and cross-family domain shifts. We present a family-conditioned, multi-tier audio tagger that combines a LoRA-finetuned Whisper encoder with a lightweight, target-speaker-aware Transformer for long-context inference and framewise prediction across tiers. To improve temporal coherence, we incorporate a simple sequence-level smoothing loss, and to enhance robustness across households, we introduce a factorized speaker-token design with a shared tier token and a learned family-specific offset, reducing family bias and promoting generalizable representations. Together, these choices enable efficient and effective infant-centered audio tagging of daylong audio recordings in home environments.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 21 references that are in the map. First posted on arXiv 12 Aug 2026.

Builds on

16 of 21 references.

References · 16

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  7. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  8. 2024LoRA-Whisper: Parameter-Efficient and Extensible Multilingual ASRcited by 35
  9. 2022Listen, Adapt, Better WER: Source-free Single-utterance Test-time Adaptation for Automatic Speech Recognitioncited by 19
  10. 2023Sparsely Shared LoRA on Whisper for Child Speech Recognitioncited by 18
  11. 2020An open-source voice type classifier for child-centered daylong recordingscited by 14
  12. 2024Towards Rehearsal-Free Multilingual ASR: A LoRA-based Case Study on Whispercited by 9
4 more
  1. 2023Towards Robust Family-Infant Audio Analysis Based on Unsupervised Pretraining of Wav2vec 2.0 on Large-Scale Unlabeled Family Audiocited by 6
  2. 2023Robust Self Supervised Speech Embeddings for Child-Adult Classification in Interactions involving Children with Autismcited by 4
  3. 2024Exploring Speech Foundation Models for Speaker Diarization in Child-Adult Dyadic Interactionscited by 4
  4. 2024Data Efficient Child-Adult Speaker Diarization with Simulated Conversationscited by 3

Built on by

Not cited yet.