A Collection of Pragmatic-Similarity Judgments over Spoken Dialog Utterances
Not on the map. Why?
Abstract (arXiv)
Automatic measures of similarity between utterances are invaluable for training speech synthesizers, evaluating machine translation, and assessing learner productions. While there exist measures for semantic similarity and prosodic similarity, there are as yet none for pragmatic similarity. To enable the training of such measures, we developed the first collection of human judgments of pragmatic similarity between utterance pairs. Each pair consisting of an utterance extracted from a recorded dialog and a re-enactment of that utterance. Re-enactments were done under various conditions designed to create a variety of degrees of similarity. Each pair was rated on a continuous scale by 6 to 9 judges. The average inter-judge correlation was as high as 0.72 for English and 0.66 for Spanish. We make this data available at https://github.com/divettemarco/PragSim .
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 19 of its 50 references that are in the map. First posted on arXiv 21 Mar 2024.
Builds on
19 of 50 references.
References · 19
- 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
- 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
- 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 190
- 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
- 2022Generative Spoken Dialogue Language Modelingcited by 125
- 2019Beyond BLEU: Training Neural Machine Translation with Semantic Similaritycited by 84
- 2020Evolution of Semantic Similarity -- A Surveycited by 83
- 2021Generalization Ability of MOS Prediction Networkscited by 69
- 2022The VoiceMOS Challenge 2022cited by 57
- 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
- 2018Learning Semantic Textual Similarity from Conversationscited by 41
- 2022SpeechLMScore: Evaluating speech generation using speech language modelcited by 19
7 more
- 2022BLASER: A Text-Free Speech-to-Speech Translation Evaluation Metriccited by 16
- 2020Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshopcited by 15
- 2021Assessing Evaluation Metrics for Speech-to-Speech Translationcited by 10
- 2023A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translationcited by 10
- 2023Automatic Evaluation of Turn-taking Cues in Conversational Speech Synthesiscited by 8
- 2023Towards cross-language prosody transfer for dialogcited by 5
- 2022A Textless Metric for Speech-to-Speech Comparisoncited by 4
Built on by
Citing papers per year: 2024: 2.
2 papers.