# EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis

arXiv: https://arxiv.org/abs/2601.22873 · HTML: https://mlsanity.com/paper/2601.22873/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability.
- Builds on 28 of its 31 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- Mechanistic interpretability / [Activation steering methods](https://mlsanity.com/mechinterp/activation-steering-methods/)

## Builds on (most cited first, 28 of 28)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Steering Llama 2 via Contrastive Activation Addition](https://mlsanity.com/paper/2312.06681/) (2023)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering](https://mlsanity.com/paper/2311.06668/) (2023)
- [DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressors](https://mlsanity.com/paper/2010.15258/) (2020)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization](https://mlsanity.com/paper/2406.00045/) (2024)
- [Emotional Voice Conversion: Theory, Databases and ESD](https://mlsanity.com/paper/2105.14762/) (2021)
- [Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categories](https://mlsanity.com/paper/2406.00034/) (2024)
- [Word Embeddings Are Steers for Language Models](https://mlsanity.com/paper/2305.12798/) (2023)
- [Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering](https://mlsanity.com/paper/2410.15999/) (2024)
- [Style Vectors for Steering Generative Large Language Model](https://mlsanity.com/paper/2402.01618/) (2024)
- [EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidance](https://mlsanity.com/paper/2211.09496/) (2022)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimization](https://mlsanity.com/paper/2409.10157/) (2024)
- [Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesis](https://mlsanity.com/paper/2011.08477/) (2020)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Survey](https://mlsanity.com/paper/2412.06602/) (2024)
- [EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering](https://mlsanity.com/paper/2508.03543/) (2025)
- [Automating Steering for Safe Multimodal Large Language Models](https://mlsanity.com/paper/2507.13255/) (2025)
- [Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesis](https://mlsanity.com/paper/2405.09171/) (2024)
- [Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis](https://mlsanity.com/paper/2505.12597/) (2025)
- [Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2412.18733/) (2024)
- [Representation Tuning](https://mlsanity.com/paper/2409.06927/) (2024)
- [PROEMO: Prompt-Driven Text-to-Speech Synthesis Based on Emotion and Intensity Control](https://mlsanity.com/paper/2501.06276/) (2025)
- [From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Test](https://mlsanity.com/paper/2505.18562/) (2025)

## Built on by (most cited first, 7 of 7)

- [The Sound of Absence: Audio-Language Embedding Models Struggle with Negation](https://mlsanity.com/paper/2607.12290/) (2026)
- [DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speech](https://mlsanity.com/paper/2606.00066/) (2026)
- [Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2609.27399/) (2026)
- [Controlling Speaking Rate in Autoregressive TTS via Activation Steering](https://mlsanity.com/paper/2609.33810/) (2026)
- [SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flows](https://mlsanity.com/paper/2609.34648/) (2026)
- [ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometry](https://mlsanity.com/paper/2610.03125/) (2026)
- [AphaVoice: Text-to-Speech Model for Aphasia Patient Simulation](https://mlsanity.com/paper/2610.14479/) (2026)

## Abstract (arXiv)

Achieving precise and controllable emotional expression is crucial for producing natural and context-appropriate speech in text-to-speech (TTS) synthesis. However, many emotion-aware TTS systems, including large language model (LLM)-based designs, rely on scaling fixed emotion embeddings or external guidance, limiting their ability to model emotion-specific latent characteristics. To address this gap, we present EmoShift, a lightweight activation-steering framework incorporating a EmoSteer layer, which learns a steering vector for each target emotion in the output embedding space to capture its latent offset and maintain stable, appropriate expression across utterances and categories. With only 10M trainable parameters,less than 1/30 of full fine-tuning, EmoShift outperforms zero-shot and fully fine-tuned baselines in objective and subjective evaluations, enhancing emotional expressiveness while preserving naturalness and speaker similarity. Further analysis confirms the proposed EmoSteer layer's effectiveness and reveals its potential for controllable emotional intensity in speech synthesis.