# How to get the best and fairest performance out of your ASR system using activation steering

arXiv: https://arxiv.org/abs/2610.26994 · HTML: https://mlsanity.com/paper/2610.26994/

## Key facts

- No citing papers in the map yet (data through 19 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 19 Oct 2026.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)
- Speech and audio language models / [ASR evaluation, bias and atypical speech](https://mlsanity.com/speech/asr-evaluation-bias-and-atypical-speech/)

## Builds on (most cited first, 24 of 24)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoders](https://mlsanity.com/paper/2501.17148/) (2025)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [Toward Fairness in Speech Recognition: Discovery and mitigation of performance disparities](https://mlsanity.com/paper/2207.11345/) (2022)
- [Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing](https://mlsanity.com/paper/2108.00917/) (2021)
- [CommonAccent: Exploring Large Acoustic Pretrained Models for Accent Classification Based on Common Voice](https://mlsanity.com/paper/2305.18283/) (2023)
- [Accent-Robust Automatic Speech Recognition Using Supervised and Unsupervised Wav2vec Embeddings](https://mlsanity.com/paper/2110.03520/) (2021)
- [Improving Fairness and Robustness in End-to-End Speech Recognition through unsupervised clustering](https://mlsanity.com/paper/2306.06083/) (2023)
- [AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoders](https://mlsanity.com/paper/2602.05027/) (2026)
- [Sonos Voice Control Bias Assessment Dataset: A Methodology for Demographic Bias Assessment in Voice Assistants](https://mlsanity.com/paper/2405.19342/) (2024)
- [Refusal Steering: Fine-grained Control over LLM Refusal Behaviour for Sensitive Topics](https://mlsanity.com/paper/2512.16602/) (2025)
- [Activation Steering for Accent Adaptation in Large Audio Language Models](https://mlsanity.com/paper/2603.05813/) (2026)
- [The Effectiveness of Style Vectors for Steering Large Language Models: A Human Evaluation](https://mlsanity.com/paper/2601.21505/) (2026)
- [Activation Steering for Accent-Neutralized Zero-Shot Text-To-Speech](https://mlsanity.com/paper/2603.05977/) (2026)
- [SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors](https://mlsanity.com/paper/2606.00460/) (2026)
- [A Study of Data Selection Strategies for Pre-training Self-Supervised Speech Models](https://mlsanity.com/paper/2601.20896/) (2026)
- [Toward Fair Speech Technologies: A Comprehensive Survey of Bias and Fairness in Speech AI](https://mlsanity.com/paper/2605.01597/) (2026)
- [Where Do Self-Supervised Speech Models Become Unfair?](https://mlsanity.com/paper/2604.18249/) (2026)
- [Responsible Benchmarking of Fairness for Automatic Speech Recognition](https://mlsanity.com/paper/2605.10615/) (2026)
- [Speaker Group Encoding in Self-supervised Speech Recognition Models](https://mlsanity.com/paper/2606.10654/) (2026)

## Abstract (arXiv)

Activation steering for ASR systems is an underexplored technique in improving both ASR performance and demographic fairness. We investigate two forms of activation steering: paired activation subtraction, and supervised optimization. We find the former to be ineffective at ameliorating ASR performance whatsoever, while the latter is quite effective at deriving steering vectors (SVs) that improve at once overall ASR performance and demographic fairness along such diverse axes as gender, age, and dialect. We also explore what types of data---from the most to the least personal---are necessary for calculating or training SVs. First, we show significant performance in performance from training SVs using an eclectic group of speakers from the same dataset. Furthermore, we find that vectors trained using acoustically similar speakers leads to ASR performances that are competitive or superior to vectors trained on the same speaker, or speakers from the same labeled demographic group. This finding shows that we can harness activation steering for few-shot adaptation given access to a speaker's demography, or zero-shot adaptation to unseen speakers without the need for any labels at all.