# Counterfactual Activation Editing for Post-hoc Prosody and Mispronunciation Correction in TTS Models

arXiv: https://arxiv.org/abs/2506.00832 · HTML: https://mlsanity.com/paper/2506.00832/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Builds on (most cited first, 24 of 24)

- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Natural TTS Synthesis by Conditioning WaveNet on Mel Spectrogram Predictions](https://mlsanity.com/paper/1712.05884/) (2017)
- [FastSpeech 2: Fast and High-Quality End-to-End Text to Speech](https://mlsanity.com/paper/2006.04558/) (2020)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Style Tokens: Unsupervised Style Modeling, Control and Transfer in End-to-End Speech Synthesis](https://mlsanity.com/paper/1803.09017/) (2018)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [What do Neural Machine Translation Models Learn about Morphology?](https://mlsanity.com/paper/1704.03471/) (2017)
- [Towards Realistic Individual Recourse and Actionable Explanations in Black-Box Decision Making Systems](https://mlsanity.com/paper/1907.09615/) (2019)
- [What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Models](https://mlsanity.com/paper/1812.09355/) (2018)
- [Hierarchical Generative Modeling for Controllable Speech Synthesis](https://mlsanity.com/paper/1810.07217/) (2018)
- [Diffeomorphic Counterfactuals with Generative Models](https://mlsanity.com/paper/2206.05075/) (2022)
- [Unified Mandarin TTS Front-end Based on Distilled BERT Model](https://mlsanity.com/paper/2012.15404/) (2020)
- [Exact Prosody Cloning in Zero-Shot Multispeaker Text-to-Speech](https://mlsanity.com/paper/2206.12229/) (2022)
- [Hierarchical prosody modeling and control in non-autoregressive parallel neural TTS](https://mlsanity.com/paper/2110.02952/) (2021)
- [Ctrl-P: Temporal Control of Prosodic Variation for Speech Synthesis](https://mlsanity.com/paper/2106.08352/) (2021)
- [Predicting within and across language phoneme recognition performance of self-supervised learning speech pre-trained models](https://mlsanity.com/paper/2206.12489/) (2022)
- [Speaking rate attention-based duration prediction for speed control TTS](https://mlsanity.com/paper/2310.08846/) (2023)

## Built on by (most cited first, 3 of 3)

- [Manifold-Aligned Guided Integrated Gradients for Reliable Feature Attribution](https://mlsanity.com/paper/2605.02167/) (2026)
- [Causal Prosody Mediation for Text-to-Speech:Counterfactual Training of Duration, Pitch, and Energy in FastSpeech2](https://mlsanity.com/paper/2603.11683/) (2026)
- [Steering Speech-Language Models: Training-Free Task Specialization via Contrastive Activation Addition](https://mlsanity.com/paper/2610.04683/) (2026)

## Abstract (arXiv)

Recent advances in Text-to-Speech (TTS) have significantly improved speech naturalness, increasing the demand for precise prosody control and mispronunciation correction. Existing approaches for prosody manipulation often depend on specialized modules or additional training, limiting their capacity for post-hoc adjustments. Similarly, traditional mispronunciation correction relies on grapheme-to-phoneme dictionaries, making it less practical in low-resource settings. We introduce Counterfactual Activation Editing, a model-agnostic method that manipulates internal representations in a pre-trained TTS model to achieve post-hoc control of prosody and pronunciation. Experimental results show that our method effectively adjusts prosodic features and corrects mispronunciations while preserving synthesis quality. This opens the door to inference-time refinement of TTS outputs without retraining, bridging the gap between pre-trained TTS models and editable speech synthesis.