# Afrispeech Semantics: Evaluating Audio Semantic Reasoning in Spoken Language Models Across Domains and Accents

arXiv: https://arxiv.org/abs/2606.11219 · HTML: https://mlsanity.com/paper/2606.11219/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 27 references that are in the map.
- First posted on arXiv 11 May 2026.

## Placed in

- LLM reasoning / [Audio-language model reasoning](https://mlsanity.com/lr/audio-language-model-reasoning/)

## Builds on (most cited first, 18 of 18)

- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [AudioBench: A Universal Benchmark for Audio Large Language Models](https://mlsanity.com/paper/2406.16020/) (2024)
- [A Survey on Speech Large Language Models for Understanding](https://mlsanity.com/paper/2410.18908/) (2024)
- [ML-SUPERB 2.0: Benchmarking Multilingual Speech Models Across Modeling Constraints, Languages, and Datasets](https://mlsanity.com/paper/2406.08641/) (2024)
- [Audio-Aware Large Language Models as Judges for Speaking Styles](https://mlsanity.com/paper/2506.05984/) (2025)
- [SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2508.02018/) (2025)
- [Training Audio Captioning Models without Audio](https://mlsanity.com/paper/2309.07372/) (2023)
- [Afrispeech-Dialog: A Benchmark Dataset for Spontaneous English Conversations in Healthcare and Beyond](https://mlsanity.com/paper/2502.03945/) (2025)
- [Attacker's Noise Can Manipulate Your Audio-based LLM in the Real World](https://mlsanity.com/paper/2507.06256/) (2025)
- [BoSS: Beyond-Semantic Speech](https://mlsanity.com/paper/2507.17563/) (2025)
- [Preservation of Language Understanding Capabilities in Speech-aware Large Language Models](https://mlsanity.com/paper/2509.12171/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Word Error Rate: A Switch Aware Evaluation of ASR and Audio Language Models on English Yoruba Code-Switched Speech](https://mlsanity.com/paper/2609.11786/) (2026)
- [How Robust Are Neural Audio Codecs for African Speech? A Multi-Task Benchmark and the Limits of Perceptual Quality](https://mlsanity.com/paper/2610.00154/) (2026)

## Abstract (arXiv)

Audio language models (ALMs) are increasingly used for speech-based understanding, yet their ability to perform semantic reasoning beyond transcription, Text-to-Audio Retrieval, Captioning, and Question-Answering accuracy remains insufficiently benchmarked. In particular, the effects of accent variation, domain shift, and semantic over-inference on audio reasoning are poorly understood. We evaluate audio language models across five semantic and paralinguistic reasoning tasks: entailment, consistency, plausibility, accent drift, and accent restraint. Collectively, these tasks assess a model's ability to reason over spoken audio as the primary evidence source, including whether a textual hypothesis can be inferred, contradicted, or left undetermined by the audio, whether statements align or conflict with spoken content, whether claims are plausible given the discourse, and whether model predictions remain stable or appropriately constrained across accent variation. These findings highlight critical limitations in current audio reasoning evaluations and hope to provide guidance for more robust and equitable ALM design and assessment