# Re-ENACT: Reinforcement Learning for Emotional Speech Generation using Actor-Critic Strategy

arXiv: https://arxiv.org/abs/2408.01892 · HTML: https://mlsanity.com/paper/2408.01892/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL training of language and agent models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 11 of its 31 references that are in the map.
- First posted on arXiv 4 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL training of language and agent models](https://mlsanity.com/deeprl/rl-training-of-language-and-agent-models/)

## Builds on (most cited first, 11 of 11)

- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [Variational Inference: A Review for Statisticians](https://mlsanity.com/paper/1601.00670/) (2016)
- [Mellotron: Multispeaker expressive voice synthesis by conditioning on rhythm, pitch and global style tokens](https://mlsanity.com/paper/1910.11997/) (2019)
- [AdaMAE: Adaptive Masking for Efficient Spatiotemporal Learning with Masked Autoencoders](https://mlsanity.com/paper/2211.09120/) (2022)
- [Transforming Spectrum and Prosody for Emotional Voice Conversion with Non-Parallel Training Data](https://mlsanity.com/paper/2002.00198/) (2020)
- [Non-parallel Emotion Conversion using a Deep-Generative Hybrid Network and an Adversarial Pair Discriminator](https://mlsanity.com/paper/2007.12932/) (2020)
- [Towards Automated Assessment of Stuttering and Stuttering Therapy](https://mlsanity.com/paper/2006.09222/) (2020)
- [Multi-speaker Emotion Conversion via Latent Variable Regularization and a Chained Encoder-Decoder-Predictor Network](https://mlsanity.com/paper/2007.12937/) (2020)
- [A Diffeomorphic Flow-based Variational Framework for Multi-speaker Emotion Conversion](https://mlsanity.com/paper/2211.05071/) (2022)

## Abstract (arXiv)

In this paper, we propose the first method to modify the prosodic features of a given speech signal using actor-critic reinforcement learning strategy. Our approach uses a Bayesian framework to identify contiguous segments of importance that links segments of the given utterances to perception of emotions in humans. We train a neural network to produce the variational posterior of a collection of Bernoulli random variables; our model applies a Markov prior on it to ensure continuity. A sample from this distribution is used for downstream emotion prediction. Further, we train the neural network to predict a soft assignment over emotion categories as the target variable. In the next step, we modify the prosodic features (pitch, intensity, and rhythm) of the masked segment to increase the score of target emotion. We employ an actor-critic reinforcement learning to train the prosody modifier by discretizing the space of modifications. Further, it provides a simple solution to the problem of gradient computation through WSOLA operation for rhythm manipulation. Our experiments demonstrate that this framework changes the perceived emotion of a given speech utterance to the target. Further, we show that our unified technique is on par with state-of-the-art emotion conversion models from supervised and unsupervised domains that require pairwise training.