# It's not what you said, it's how you said it: discriminative perception of speech as a multichannel communication system

arXiv: https://arxiv.org/abs/2105.00260 · HTML: https://mlsanity.com/paper/2105.00260/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 41 references that are in the map.
- First posted on arXiv 1 May 2021.

## Builds on (most cited first, 7 of 7)

- [Unsupervised speech representation learning using WaveNet autoencoders](https://mlsanity.com/paper/1901.08810/) (2019)
- [Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trends](https://mlsanity.com/paper/2001.00378/) (2020)
- [Multimodal Continuous Turn-Taking Prediction Using Multiscale RNNs](https://mlsanity.com/paper/1808.10785/) (2018)
- [Is this Dialogue Coherent? Learning from Dialogue Acts and Entities](https://mlsanity.com/paper/2006.10157/) (2020)
- [Coherence Models for Dialogue](https://mlsanity.com/paper/1806.08044/) (2018)
- [Unspeech: Unsupervised Speech Context Embeddings](https://mlsanity.com/paper/1804.06775/) (2018)
- [Perception of prosodic variation for speech synthesis using an unsupervised discrete representation of F0](https://mlsanity.com/paper/2003.06686/) (2020)

## Built on by (most cited first, 6 of 6)

- [A neural prosody encoder for end-ro-end dialogue act classification](https://mlsanity.com/paper/2205.05590/) (2022)
- [Improving End-to-End SLU performance with Prosodic Attention and Distillation](https://mlsanity.com/paper/2305.08067/) (2023)
- [Quantifying the perceptual value of lexical and non-lexical channels in speech](https://mlsanity.com/paper/2307.03534/) (2023)
- [Joint Learning of Context and Feedback Embeddings in Spoken Dialogue](https://mlsanity.com/paper/2406.07291/) (2024)
- [Real-Time Voice AI Hears but Does Not Listen](https://mlsanity.com/paper/2606.26083/) (2026)
- [Reading the Mood Behind Words: Integrating Prosody-Derived Emotional Context into Socially Responsive VR Agents](https://mlsanity.com/paper/2603.09324/) (2026)

## Abstract (arXiv)

People convey information extremely effectively through spoken interaction using multiple channels of information transmission: the lexical channel of what is said, and the non-lexical channel of how it is said. We propose studying human perception of spoken communication as a means to better understand how information is encoded across these channels, focusing on the question 'What characteristics of communicative context affect listener's expectations of speech?'. To investigate this, we present a novel behavioural task testing whether listeners can discriminate between the true utterance in a dialogue and utterances sampled from other contexts with the same lexical content. We characterize how perception - and subsequent discriminative capability - is affected by different degrees of additional contextual information across both the lexical and non-lexical channel of speech. Results demonstrate that people can effectively discriminate between different prosodic realisations, that non-lexical context is informative, and that this channel provides more salient information than the lexical channel, highlighting the importance of the non-lexical channel in spoken interaction.