# What Do Humans Hear When Interacting? Experiments on Selective Listening for Evaluating ASR of Spoken Dialogue Systems

arXiv: https://arxiv.org/abs/2508.04402 · HTML: https://mlsanity.com/paper/2508.04402/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 29 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 11 of 11)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Achieving Human Parity in Conversational Speech Recognition](https://mlsanity.com/paper/1610.05256/) (2016)
- [Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalization](https://mlsanity.com/paper/2305.11095/) (2023)
- [Semantic Distance: A New Metric for ASR Performance Analysis Towards Spoken Language Understanding](https://mlsanity.com/paper/2104.02138/) (2021)
- [Retrieval Augmented End-to-End Spoken Dialog Models](https://mlsanity.com/paper/2402.01828/) (2024)
- [Evaluating the Usability of Automatically Generated Captions for People who are Deaf or Hard of Hearing](https://mlsanity.com/paper/1712.02033/) (2017)
- [H_eval: A new hybrid evaluation metric for automatic speech recognition tasks](https://mlsanity.com/paper/2211.01722/) (2022)
- [Perceptimatic: A human speech perception benchmark for unsupervised subword modelling](https://mlsanity.com/paper/2010.05961/) (2020)
- [A Transformer-based Response Evaluator for Open-Domain Spoken Conversation](https://mlsanity.com/paper/2302.04424/) (2023)
- [Dialog act guided contextual adapter for personalized speech recognition](https://mlsanity.com/paper/2303.17799/) (2023)

## Abstract (arXiv)

Spoken dialogue systems (SDSs) utilize automatic speech recognition (ASR) at the front end of their pipeline. The role of ASR in SDSs is to recognize information in user speech related to response generation appropriately. Examining selective listening of humans, which refers to the ability to focus on and listen to important parts of a conversation during the speech, will enable us to identify the ASR capabilities required for SDSs and evaluate them. In this study, we experimentally confirmed selective listening when humans generate dialogue responses by comparing human transcriptions for generating dialogue responses and reference transcriptions. Based on our experimental results, we discuss the possibility of a new ASR evaluation method that leverages human selective listening, which can identify the gap between transcription ability between ASR systems and humans.