mlsanityData to 14 Oct 2026

EmoShift: Lightweight Activation Steering for Enhanced Emotion-Aware Speech Synthesis

arXiv 2601.22873 ↗First posted 30 Jan 2026eess.AS cs.AI cs.CLcited by 7

On the map: Mechanistic interpretability → Activation steering methods

Abstract (arXiv)

Achieving precise and controllable emotional expression is crucial for producing natural and context-appropriate speech in text-to-speech (TTS) synthesis. However, many emotion-aware TTS systems, including large language model (LLM)-based designs, rely on scaling fixed emotion embeddings or external guidance, limiting their ability to model emotion-specific latent characteristics. To address this gap, we present EmoShift, a lightweight activation-steering framework incorporating a EmoSteer layer, which learns a steering vector for each target emotion in the output embedding space to capture its latent offset and maintain stable, appropriate expression across utterances and categories. With only 10M trainable parameters,less than 1/30 of full fine-tuning, EmoShift outperforms zero-shot and fully fine-tuned baselines in objective and subjective evaluations, enhancing emotional expressiveness while preserving naturalness and speaker similarity. Further analysis confirms the proposed EmoSteer layer's effectiveness and reveals its potential for controllable emotional intensity in speech synthesis.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Activation steering methods, one of 13 directions in Mechanistic interpretability. Builds on 28 of its 31 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

28 of 31 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025EmoSteer-TTS: Fine-Grained and Training-Free Emotion-Controllable Text-to-Speech via Activation Steering5 Aug
  2. 2024Improving Instruction-Following in Language Models through Activation Steering15 Oct
  3. 2024Personalized Steering of Large Language Models: Versatile Steering Vectors Through Bi-directional Preference Optimization28 May
  4. 2023Steering Llama 2 via Contrastive Activation Addition9 Dec

Other references · 24

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 799
  3. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  4. 2023In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steeringcited by 245
  5. 2020DNSMOS: A Non-Intrusive Perceptual Objective Speech Quality metric to evaluate Noise Suppressorscited by 241
  6. 2023emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representationcited by 205
  7. 2021Emotional Voice Conversion: Theory, Databases and ESDcited by 115
  8. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 77
  9. 2023Word Embeddings Are Steers for Language Modelscited by 70
  10. 2024Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineeringcited by 63
  11. 2024Style Vectors for Steering Generative Large Language Modelcited by 56
  12. 2022EmoDiff: Intensity Controllable Emotional Text-to-Speech with Soft-Label Guidancecited by 37
12 more
  1. 2025EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Promptingcited by 34
  2. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
  3. 2020Fine-grained Emotion Strength Transfer, Control and Prediction for Emotional Speech Synthesiscited by 29
  4. 2024EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speechcited by 24
  5. 2024Towards Controllable Speech Synthesis in the Era of Large Language Models: A Systematic Surveycited by 23
  6. 2025Automating Steering for Safe Multimodal Large Language Modelscited by 17
  7. 2024Hierarchical Emotion Prediction and Control in Text-to-Speech Synthesiscited by 7
  8. 2025Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesiscited by 7
  9. 2024Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesiscited by 5
  10. 2024Representation Tuningcited by 3
  11. 2025PROEMO: Prompt-Driven Text-to-Speech Synthesis Based on Emotion and Intensity Controlcited by 2
  12. 2025From Word to World: Evaluate and Mitigate Culture Bias in LLMs via Word Association Testcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 7.

7 papers.

  1. 2026The Sound of Absence: Audio-Language Embedding Models Struggle with Negationcited by 1
  2. 2026DUET: Unified Dual-Space Emotion Control for Diffusion and Flow-Matching Driven Text-to-Speechcited by 0
  3. 2026Forget who you Forgot: Speaker Unlearning to Prevent Re-Identification in Zero-Shot Text-to-Speechcited by 0
  4. 2026Controlling Speaking Rate in Autoregressive TTS via Activation Steeringcited by 0
  5. 2026SEmoEdit: Probing and Harnessing the Editability of Pre-trained Speech Flowscited by 0
  6. 2026ParaGeo: Decomposing Paralinguistic Variation into a Shared Latent Geometrycited by 0
  7. 2026AphaVoice: Text-to-Speech Model for Aphasia Patient Simulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Steerspeech: Activation Steering For Emotion Control In Generated Speech
  2. 2026EmoRES-TTS: Residual-Enhanced Vector Steering for Emotional Speech Generation
  3. 2026CoCoEmo: Composable and Controllable Human-Like Emotional TTS via Activation Steering
  4. 2026A Geometric Perspective on Composable Emotion Steering in Text-to-Speech Models
  5. 2024Hierarchical Control of Emotion Rendering in Speech Synthesis
  6. 2026Sparse Autoencoders for Interpretable Emotion Control in Text-to-Speech
  7. 2026TRACE-EVC: Text-Guided Relative Affective Control for Zero-Shot Emotional Voice Conversion ↗
  8. 2026EMODY Flow: Emotion-Aware Audio-Driven Full-Body Motion Generation