# Geometric Alignment without Functional Equivalence: A Layer-wise Analysis of the Speech-Text Modality Gap

arXiv: https://arxiv.org/abs/2603.01502 · HTML: https://mlsanity.com/paper/2603.01502/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 30 references that are in the map.
- First posted on arXiv 2 Mar 2026.

## Builds on (most cited first, 28 of 28)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [The Linear Representation Hypothesis and the Geometry of Large Language Models](https://mlsanity.com/paper/2311.03658/) (2023)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Audio Flamingo 3: Advancing Audio Intelligence with Fully Open Large Audio Language Models](https://mlsanity.com/paper/2507.08128/) (2025)
- [VoiceBench: Benchmarking LLM-Based Voice Assistants](https://mlsanity.com/paper/2410.17196/) (2024)
- [LLaMA-Omni: Seamless Speech Interaction with Large Language Models](https://mlsanity.com/paper/2409.06666/) (2024)
- [The Remarkable Robustness of LLMs: Stages of Inference?](https://mlsanity.com/paper/2406.19384/) (2024)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [Semantics at an Angle: When Cosine Similarity Works Until It Doesn't](https://mlsanity.com/paper/2504.16318/) (2025)
- [Beyond Transcription: Mechanistic Interpretability in ASR](https://mlsanity.com/paper/2508.15882/) (2025)
- [Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models](https://mlsanity.com/paper/2510.12116/) (2025)
- [When Large Language Models Meet Speech: A Survey on Integration Approaches](https://mlsanity.com/paper/2502.19548/) (2025)
- [Closing the Modality Reasoning Gap for Speech Large Language Models](https://mlsanity.com/paper/2601.05543/) (2026)
- [Speech Token Prediction via Compressed-to-fine Language Modeling for Speech Generation](https://mlsanity.com/paper/2505.24496/) (2025)
- [S2SBench: A Benchmark for Quantifying Intelligence Degradation in Speech-to-Speech Large Language Models](https://mlsanity.com/paper/2505.14438/) (2025)
- [ALAS: An Automatic Latent Alignment Score for Audio Language Models](https://mlsanity.com/paper/2505.19937/) (2025)
- [How Redundant Is the Transformer Stack in Speech Representation Models?](https://mlsanity.com/paper/2409.16302/) (2024)
- [Entropy-based Coarse and Compressed Semantic Speech Representation Learning](https://mlsanity.com/paper/2509.00503/) (2025)

## Built on by (most cited first, 7 of 7)

- [Minimizing Modality Gap from the Input Side: Your Speech LLM Can Be a Prosody-Aware Text LLM](https://mlsanity.com/paper/2605.05927/) (2026)
- [Entity Binding Failures in Speech LLM Reasoning: Diagnosis and Chain-of-Thought Intervention](https://mlsanity.com/paper/2606.04474/) (2026)
- [How to Leverage Synthetic Speech for LLM-Based ASR Systems?](https://mlsanity.com/paper/2606.29031/) (2026)
- [Quantifying the Generation Modality Gap in Speech-Text Language Models](https://mlsanity.com/paper/2609.14743/) (2026)
- [Interleaved Speech Language Models Latently Work In Text](https://mlsanity.com/paper/2606.22473/) (2026)
- [Encoder-Side Neuron Identification and Amplification for Acoustic Perception in Large Audio-Language Models](https://mlsanity.com/paper/2607.11801/) (2026)
- [Repairing Speech Access After Text Updates with Frozen Affine Fields](https://mlsanity.com/paper/2610.20253/) (2026)

## Abstract (arXiv)

End-to-end speech large language models (LLMs) often answer the same question less accurately from speech than from text, even when the spoken content matches the text prompt. Existing analyses study this modality gap through the geometry of representations and measure how closely speech representations align with text. Geometric alignment gives the gap a measurable form, but it does not show whether the model uses aligned representations in the same way. This work connects representation geometry with answer formation and asks whether geometric alignment implies functional equivalence. We compare cross-layer Centered Kernel Alignment (CKA) with layer-wise answer margins for four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH. In Qwen2.5-Omni and MiniCPM-o, speech and text representations align in middle layers while their late-layer answer margins still differ, so mid-layer alignment does not imply functional equivalence. Qwen2-Audio instead shows weak alignment, which indicates a different failure mode. An attention intervention at fixed input and token count identifies a causal effect of late attention allocation on answer-code separation, a quantity distinct from answer correctness. Compressing the speech signal or its token sequence repairs a subset of the errors that occur only with speech.