# SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

arXiv: https://arxiv.org/abs/2606.00460 · HTML: https://mlsanity.com/paper/2606.00460/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 30 May 2026.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 19 of 19)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization](https://mlsanity.com/paper/2406.00045/) (2024)
- [A Unified Understanding and Evaluation of Steering Methods](https://mlsanity.com/paper/2502.02716/) (2025)
- [Can Whisper perform speech-based in-context learning?](https://mlsanity.com/paper/2309.07081/) (2023)
- [My Science Tutor (MyST) -- A Large Corpus of Children's Conversational Speech](https://mlsanity.com/paper/2309.13347/) (2023)
- [Granite-speech: open-source speech-aware LLMs with strong English ASR capabilities](https://mlsanity.com/paper/2505.08699/) (2025)
- [Learning to Steer: Input-dependent Steering for Multimodal LLMs](https://mlsanity.com/paper/2508.12815/) (2025)
- [In-Context Learning Boosts Speech Recognition via Human-like Adaptation to Speakers and Language Varieties](https://mlsanity.com/paper/2505.14887/) (2025)
- [Advancing Speech Understanding in Speech-Aware Language Models with GRPO](https://mlsanity.com/paper/2509.16990/) (2025)
- [TICL+: A Case Study On Speech In-Context Learning for Children's Speech Recognition](https://mlsanity.com/paper/2512.18263/) (2025)
- [Activation Steering for Accent Adaptation in Large Audio Language Models](https://mlsanity.com/paper/2603.05813/) (2026)
- [Steer-MoE: Efficient Audio-Language Alignment with a Mixture-of-Experts Steering Module](https://mlsanity.com/paper/2510.13558/) (2025)

## Built on by (most cited first, 4 of 4)

- [Latent-IM: Latent Interaction Management for Speech LLMs](https://mlsanity.com/paper/2607.26928/) (2026)
- [Controlling Speaking Rate in Autoregressive TTS via Activation Steering](https://mlsanity.com/paper/2609.33810/) (2026)
- [Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition](https://mlsanity.com/paper/2610.04683/) (2026)
- [Repairing Speech Access After Text Updates with Frozen Affine Fields](https://mlsanity.com/paper/2610.20253/) (2026)

## Abstract (arXiv)

Speech-aware large language models often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained language model representation space, rather than by modifying the decoder itself.