# Phonemes vs. Projectors: An Investigation of Speech-Language Interfaces for LLM-based ASR

arXiv: https://arxiv.org/abs/2604.09332 · HTML: https://mlsanity.com/paper/2604.09332/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Apr 2026.

## Built on by (most cited first, 2 of 2)

- [Latent Softmax for Data-Efficient Phoneme-Based Multilingual ASR Across Tonal and Non-Tonal Languages](https://mlsanity.com/paper/2608.01281/) (2026)
- [Phoneme-Guided Initialization for LLM-based Speech Recognition](https://mlsanity.com/paper/2610.08994/) (2026)

## Abstract (arXiv)

Integrating pretrained speech encoders with large language models (LLMs) is promising for ASR, but performance and data efficiency depend on the speech-language interface. A common choice is a learned projector that maps encoder features into the LLM embedding space, whereas an alternative is to expose discrete phoneme sequences to the LLM. Using the same encoder and LLM backbones, we compare phoneme-based and vanilla projector-based interfaces in high-resource English and low-resource Tatar. We also propose a BPE-phoneme interface that groups frequent local phoneme patterns while preserving explicit word-boundary cues for phoneme-to-grapheme generation. On LibriSpeech, the phoneme-based interface is competitive with the vanilla projector, and the BPE-phoneme interface yields further gains. On Tatar, the phoneme-based interface substantially outperforms the vanilla projector. We further find that phoneme supervision yields a phoneme-informed hybrid interface that is stronger than the vanilla projector.