# LLM-Based Multi-Reference Evaluation for Efficient and Robust Assessment of Phrase Break Annotations

arXiv: https://arxiv.org/abs/2606.21098 · HTML: https://mlsanity.com/paper/2606.21098/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 19 Jun 2026.

## Builds on (most cited first, 17 of 17)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Large Language Models Are State-of-the-Art Evaluators of Translation Quality](https://mlsanity.com/paper/2302.14520/) (2023)
- [Batch Prompting: Efficient Inference with Large Language Model APIs](https://mlsanity.com/paper/2301.08721/) (2023)
- [On Learning to Summarize with Large Language Models as References](https://mlsanity.com/paper/2305.14239/) (2023)
- [RevisEval: Improving LLM-as-a-Judge via Response-Adapted References](https://mlsanity.com/paper/2410.05193/) (2024)
- [Phrase break prediction with bidirectional encoder representations in Japanese text-to-speech synthesis](https://mlsanity.com/paper/2104.12395/) (2021)
- [Not All Metrics Are Guilty: Improving NLG Evaluation by Diversifying References](https://mlsanity.com/paper/2305.15067/) (2023)
- [Modeling Prosodic Phrasing with Multi-Task Learning in Tacotron-based TTS](https://mlsanity.com/paper/2008.05284/) (2020)
- [Duration-aware pause insertion using pre-trained language model for multi-speaker text-to-speech](https://mlsanity.com/paper/2302.13652/) (2023)
- [Assessing Phrase Break of ESL Speech with Pre-trained Language Models and Large Language Models](https://mlsanity.com/paper/2306.04980/) (2023)
- [PauseSpeech: Natural Speech Synthesis via Pre-trained Language Model and Pause-based Prosody Modeling](https://mlsanity.com/paper/2306.07489/) (2023)
- [Cross-Lingual Transfer Learning for Phrase Break Prediction with Multilingual Language Model](https://mlsanity.com/paper/2306.02579/) (2023)
- [ProsodyFM: Unsupervised Phrasing and Intonation Control for Intelligible Speech Synthesis](https://mlsanity.com/paper/2412.11795/) (2024)
- [Synthetic Data Generation for Phrase Break Prediction with Large Language Model](https://mlsanity.com/paper/2507.18044/) (2025)
- [A Linguistically Motivated Analysis of Intonational Phrasing in Text-to-Speech Systems: Revealing Gaps in Syntactic Sensitivity](https://mlsanity.com/paper/2505.22236/) (2025)

## Abstract (arXiv)

Reliable evaluation of phrase break annotations is crucial, as subtle variations in prosodic boundaries directly affect the clarity and naturalness of speech. However, existing approaches exhibit major limitations: single-reference evaluation assumes a unique gold phrasing for an utterance despite multiple valid phrasings, while human judgment, though flexible, is labor-intensive and unscalable. To address these, we propose LLM-based Multi-Reference Evaluation (LMRE) for phrase break annotations that models the one-to-many nature of prosodic phrasing and generates multiple valid phrasings from minimal demonstrations. On a Korean testbed of 1,356 annotations covering five strategies, LMRE shows stronger alignment with human judgment than single-reference evaluation in both acceptance behavior and score correlation. Our findings demonstrate that LMRE effectively achieves both scalability and multi-reference support, highlighting the potential of LLMs for evaluation in the speech domain.