# Component-Aware Differential Privacy for Federated Multilingual Speech-LLMs

arXiv: https://arxiv.org/abs/2609.11762 · HTML: https://mlsanity.com/paper/2609.11762/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 18 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [TinyLlama: An Open-Source Small Language Model](https://mlsanity.com/paper/2401.02385/) (2024)
- [MLS: A Large-Scale Multilingual Dataset for Speech Research](https://mlsanity.com/paper/2012.03411/) (2020)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [An Embarrassingly Simple Approach for LLM with Strong ASR Capacity](https://mlsanity.com/paper/2402.08846/) (2024)
- [Recent Advances in Speech Language Models: A Survey](https://mlsanity.com/paper/2410.03751/) (2024)
- [Voxtral](https://mlsanity.com/paper/2507.13264/) (2025)
- [Enabling Differentially Private Federated Learning for Speech Recognition: Benchmarks, Adaptive Optimizers and Gradient Clipping](https://mlsanity.com/paper/2310.00098/) (2023)
- [What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study](https://mlsanity.com/paper/2506.12537/) (2025)

## Abstract (arXiv)

Per-layer differential privacy (DP) clipping improves gradient fidelity in federated learning by allocating per-matrix clipping budgets proportional to parameter count. We show that this recipe breaks for speech large language models (speech-LLMs), when the acoustic encoder and the language decoder differ by an order of magnitude in update norm. Single-pool per-layer methods suffer cross-component budget collapse, dragging word error rate (WER) far from flat global clipping or collapsing training entirely. When the norm imbalance is milder, adaptive single-pool methods partially recover, confirming that collapse severity scales with the inter-component norm ratio. We empirically diagnose the root cause across six per-layer methods and three speech-LLM architectures. We then propose α-split, a two-pool allocation that normalises encoder and LLM parameters into independent pools, and show that joint ℓ2 sensitivity and the original (varepsilon,δ)-DP guarantee are unchanged. At architecture-calibrated α, our method recovers WER utility compared to flat DP, while granting the encoder 4.47× tighter per-component noise protection against speaker voice-based gradient-inversion attacks at only +2.6% LLM noise overhead.