mlsanityData to 13 Oct 2026

Distinct social-linguistic processing between humans and large audio-language models: Evidence from model-brain alignment

arXiv 2503.19586 ↗First posted 25 Mar 2025cs.CL q-bio.NCcited by 3

Not on the map. Why?

Abstract (arXiv)

Voice-based AI development faces unique challenges in processing both linguistic and paralinguistic information. This study compares how large audio-language models (LALMs) and humans integrate speaker characteristics during speech comprehension, asking whether LALMs process speaker-contextualized language in ways that parallel human cognitive mechanisms. We compared two LALMs' (Qwen2-Audio and Ultravox 0.5) processing patterns with human EEG responses. Using surprisal and entropy metrics from the models, we analyzed their sensitivity to speaker-content incongruency across social stereotype violations (e.g., a man claiming to regularly get manicures) and biological knowledge violations (e.g., a man claiming to be pregnant). Results revealed that Qwen2-Audio exhibited increased surprisal for speaker-incongruent content and its surprisal values significantly predicted human N400 responses, while Ultravox 0.5 showed limited sensitivity to speaker characteristics. Importantly, neither model replicated the human-like processing distinction between social violations (eliciting N400 effects) and biological violations (eliciting P600 effects). These findings reveal both the potential and limitations of current LALMs in processing speaker-contextualized language, and suggest differences in social-linguistic processing mechanisms between humans and LALMs.

Key facts. Cited by 3 papers in the map (data through 13 Oct 2026). Builds on 10 of its 28 references that are in the map. First posted on arXiv 25 Mar 2025.

Builds on

10 of 28 references.

References · 10

  1. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  2. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 350
  3. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 335
  4. 2024Gender Bias in Large Language Models across Multiple Languagescited by 35
  5. 2024The LLM Language Network: A Neuroscientific Approach for Identifying Causally Task-Relevant Unitscited by 34
  6. 2025From Language to Cognition: How LLMs Outgrow the Human Language Networkcited by 30
  7. 2024Can Large Language Models Understand Context?cited by 26
  8. 2024Speaker effects in language comprehension: An integrative model of language and speaker processingcited by 4
  9. 2024When a Man Says He Is Pregnant: Event-related Potential Evidence for a Rational Account of Speaker-contextualized Language Comprehensioncited by 3
  10. 2025Probabilistic adaptation of language comprehension for individual speakers: evidence from neural oscillationscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Reading Between the Prompts: How Stereotypes Shape LLM's Implicit Personalizationcited by 24
  2. 2025Universal Acoustic Adversarial Attacks for Flexible Control of Speech-LLMscited by 5
  3. 2026Do Models Hear Like Us? Probing the Representational Alignment of Audio LLMs and Naturalistic EEGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Divergences between Language Models and Human Brains
  2. 2026RAIL: Rethinking Auditory Intelligence in Large Audio-Language Models with a CHC-Grounded Benchmark
  3. 2025Large Language Models Are Human-Like Internally
  4. 2026SpeechEQ: Benchmarking Emotional Intelligence Quotient in Socially Aware Voice Conversational Models
  5. 2026Interpreting Brain Responses to Language with Sparse Features from Language Models
  6. 2025SpeechIQ: Speech-Agentic Intelligence Quotient Across Cognitive Levels in Voice Understanding by Large Language Models
  7. 2023Instruction-tuning Aligns LLMs to the Human Brain
  8. 2025Scaling and context steer LLMs along the same computational path as the human brain