# The Model Hears You: Audio Language Model Deployments Should Consider the Principle of Least Privilege

arXiv: https://arxiv.org/abs/2503.16833 · HTML: https://mlsanity.com/paper/2503.16833/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 48 references that are in the map.
- First posted on arXiv 21 Mar 2025.

## Builds on (most cited first, 10 of 10)

- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Visual Adversarial Examples Jailbreak Aligned Large Language Models](https://mlsanity.com/paper/2306.13213/) (2023)
- [Pengi: An Audio Language Model for Audio Tasks](https://mlsanity.com/paper/2305.11834/) (2023)
- [AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension](https://mlsanity.com/paper/2402.07729/) (2024)
- [On Evaluating the Durability of Safeguards for Open-Weight LLMs](https://mlsanity.com/paper/2412.07097/) (2024)
- [What's in a Name? Auditing Large Language Models for Race and Gender Bias](https://mlsanity.com/paper/2402.14875/) (2024)
- [AdvWave: Stealthy Adversarial Jailbreak Attack against Large Audio-Language Models](https://mlsanity.com/paper/2412.08608/) (2024)
- [Privacy-preserving Voice Analysis via Disentangled Representations](https://mlsanity.com/paper/2007.15064/) (2020)
- [Offline Model Guard: Secure and Private ML on Mobile Devices](https://mlsanity.com/paper/2007.02351/) (2020)
- [Safeguarding Privacy in Edge Speech Understanding with Tiny Foundation Models](https://mlsanity.com/paper/2502.01649/) (2025)

## Built on by (most cited first, 2 of 2)

- [No More, No Less: Least-Privilege Language Models](https://mlsanity.com/paper/2601.23157/) (2026)
- [When Agents Act Unwatched: The Reduced-Supervision Paradox in Agentic AI](https://mlsanity.com/paper/2609.29547/) (2026)

## Abstract (arXiv)

The latest Audio Language Models (Audio LMs) process speech directly instead of relying on a separate transcription step. This shift preserves detailed information, such as intonation or the presence of multiple speakers, that would otherwise be lost in transcription. However, it also introduces new safety risks, including the potential misuse of speaker identity cues and other sensitive vocal attributes, which could have legal implications. In this paper, we urge a closer examination of how these models are built and deployed. Our experiments show that end-to-end modeling, compared with cascaded pipelines, creates socio-technical safety risks such as identity inference, biased decision-making, and emotion detection. This raises concerns about whether Audio LMs store voiceprints and function in ways that create uncertainty under existing legal regimes. We then argue that the Principle of Least Privilege should be considered to guide the development and deployment of these models. Specifically, evaluations should assess (1) the privacy and safety risks associated with end-to-end modeling; and (2) the appropriate scope of information access. Finally, we highlight related gaps in current audio LM benchmarks and identify key open research questions, both technical and policy-related, that must be addressed to enable the responsible deployment of end-to-end Audio LMs.