# HaikuS2S: A Cascaded System For Responding In Verse

arXiv: https://arxiv.org/abs/2609.23951 · HTML: https://mlsanity.com/paper/2609.23951/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 19 references that are in the map.
- First posted on arXiv 20 Sep 2026.

## Builds on (most cited first, 16 of 16)

- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Gradio: Hassle-Free Sharing and Testing of ML Models in the Wild](https://mlsanity.com/paper/1906.02569/) (2019)
- [Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotron](https://mlsanity.com/paper/1803.09047/) (2018)
- [LibriTTS-R: A Restored Multi-Speaker Text-to-Speech Corpus](https://mlsanity.com/paper/2305.18802/) (2023)
- [ESPnet-TTS: Unified, Reproducible, and Integratable Open Source End-to-End Text-to-Speech Toolkit](https://mlsanity.com/paper/1910.10909/) (2019)
- [VERSA: A Versatile Evaluation Toolkit for Speech, Audio, and Music](https://mlsanity.com/paper/2412.17667/) (2024)
- [MsEmoTTS: Multi-scale emotion transfer, prediction, and control for emotional speech synthesis](https://mlsanity.com/paper/2201.06460/) (2022)
- [OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaning](https://mlsanity.com/paper/2506.00338/) (2025)
- [ESPnet-Codec: Comprehensive Training and Evaluation of Neural Codecs for Audio, Music, and Speech](https://mlsanity.com/paper/2409.15897/) (2024)
- [iEmoTTS: Toward Robust Cross-Speaker Emotion Transfer and Control for Speech Synthesis based on Disentanglement between Prosody and Timbre](https://mlsanity.com/paper/2206.14866/) (2022)
- [ESPnet-SDS: Unified Toolkit and Demo for Spoken Dialogue Systems](https://mlsanity.com/paper/2503.08533/) (2025)
- [Prosody-TTS: An end-to-end speech synthesis system with prosody control](https://mlsanity.com/paper/2110.02854/) (2021)
- [Simple and Effective Multi-sentence TTS with Expressive and Coherent Prosody](https://mlsanity.com/paper/2206.14643/) (2022)
- [PoeticTTS -- Controllable Poetry Reading for Literary Studies](https://mlsanity.com/paper/2207.05549/) (2022)

## Abstract (arXiv)

Expressive speech synthesis has advanced through prosody modeling, yet generating structured poetic speech, such as haiku, remains challenging. Prior work on prosody transfer improves expressiveness, and fine-tuned poetry TTS (text-to-speech) systems capture verse intonation. However, these models do not model haiku's 5-7-5 syllable structure or line-ending pauses. We present a cascaded system, HaikuS2S, combining ASR (automatic speech recognition), LLM (large language model)-generated haiku, and TTS fine-tuning on both prose and custom haiku datasets. Our evaluation focuses on emotion similarity, speech quality, and prosody alignment. In our experiments, we see that our prosody and tonal alignment improve significantly with our fine-tuned systems, particularly the one trained on both general poetry and haiku. We also see that we maintain similar emotion similarity scores across all systems.