# EmphAssess : a Prosodic Benchmark on Assessing Emphasis Transfer in Speech-to-Speech Models

arXiv: https://arxiv.org/abs/2312.14069 · HTML: https://mlsanity.com/paper/2312.14069/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 21 Dec 2023.

## Builds on (most cited first, 23 of 23)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [AudioPaLM: A Large Language Model That Can Speak and Listen](https://mlsanity.com/paper/2306.12925/) (2023)
- [Generative Spoken Language Modeling from Raw Audio](https://mlsanity.com/paper/2102.01192/) (2021)
- [WhisperX: Time-Accurate Speech Transcription of Long-Form Audio](https://mlsanity.com/paper/2303.00747/) (2023)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [Textless Speech-to-Speech Translation on Real Data](https://mlsanity.com/paper/2112.08352/) (2021)
- [Text-Free Prosody-Aware Generative Spoken Language Modeling](https://mlsanity.com/paper/2109.03264/) (2021)
- [Direct speech-to-speech translation with a sequence-to-sequence model](https://mlsanity.com/paper/1904.06037/) (2019)
- [EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis](https://mlsanity.com/paper/2308.05725/) (2023)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [On the Utility of Self-supervised Models for Prosody-related Tasks](https://mlsanity.com/paper/2210.07185/) (2022)
- [Probing phoneme, language and speaker information in unsupervised speech representations](https://mlsanity.com/paper/2203.16193/) (2022)
- [ProsAudit, a prosodic benchmark for self-supervised speech models](https://mlsanity.com/paper/2302.12057/) (2023)
- [Prosodic Prominence and Boundaries in Sequence-to-Sequence Speech Synthesis](https://mlsanity.com/paper/2006.15967/) (2020)
- [IQDUBBING: Prosody modeling based on discrete self-supervised speech representation for expressive voice conversion](https://mlsanity.com/paper/2201.00269/) (2022)
- [Deep Learning For Prominence Detection In Children's Read Speech](https://mlsanity.com/paper/2110.14273/) (2021)
- [Towards cross-language prosody transfer for dialog](https://mlsanity.com/paper/2307.04123/) (2023)

## Built on by (most cited first, 22 of 22)

- [Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2505.15957/) (2025)
- [AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation](https://mlsanity.com/paper/2507.12705/) (2025)
- [Thinking with Sound: Audio Chain-of-Thought Enables Multimodal Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2509.21749/) (2025)
- [Now You Hear Me: Audio Narrative Attacks Against Large Audio-Language Models](https://mlsanity.com/paper/2601.23255/) (2026)
- [WHISTRESS: Enriching Transcriptions with Sentence Stress Detection](https://mlsanity.com/paper/2505.19103/) (2025)
- [StressTest: Can YOUR Speech LM Handle the Stress?](https://mlsanity.com/paper/2505.22765/) (2025)
- [Can LLMs Understand the Implication of Emphasized Sentences in Dialogue?](https://mlsanity.com/paper/2406.11065/) (2024)
- [LightHouse: A Survey of AGI Hallucination](https://mlsanity.com/paper/2401.06792/) (2024)
- [Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Information](https://mlsanity.com/paper/2505.15667/) (2025)
- [ProsodyLM: Uncovering the Emerging Prosody Processing Capabilities in Speech Language Models](https://mlsanity.com/paper/2507.20091/) (2025)
- [Learning Fine-Grained Controllability on Speech Generation via Efficient Fine-Tuning](https://mlsanity.com/paper/2406.06251/) (2024)
- [Benchmarking Prosody Encoding in Discrete Speech Tokens](https://mlsanity.com/paper/2508.11224/) (2025)
- [Knowing What to Stress: A Discourse-Conditioned Text-to-Speech Benchmark](https://mlsanity.com/paper/2604.10580/) (2026)
- [STEB: A Speech-to-Speech Translation Expressiveness Benchmark for Evaluating Beyond Translation Fidelity](https://mlsanity.com/paper/2606.25529/) (2026)
- [Emphasis Sensitivity in Speech Representations](https://mlsanity.com/paper/2508.11566/) (2025)
- [Centering Emotion Hotspots: Multimodal Local-Global Fusion and Cross-Modal Alignment for Emotion Recognition in Conversations](https://mlsanity.com/paper/2510.08606/) (2025)
- [Evaluating and Preserving Lexical Stress in English-to-Chinese Speech-to-Speech Translation](https://mlsanity.com/paper/2606.15266/) (2026)
- [ParaPairAudioBench: Paralinguistic Pairwise Audio Benchmark for LALM-as-a-Judge](https://mlsanity.com/paper/2606.24648/) (2026)
- [Do Speech Emphasis Models Generalize across Languages and Emotions?](https://mlsanity.com/paper/2606.27717/) (2026)
- [Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuning](https://mlsanity.com/paper/2607.02214/) (2026)
- [Iterative Self-Learning for Expressive Text-to-Speech Synthesis](https://mlsanity.com/paper/2608.15910/) (2026)
- [A Novel Sentence Stress Detection Framework Leveraging Auxiliary Word-Stress Modeling and Loss Optimization](https://mlsanity.com/paper/2610.07626/) (2026)

## Abstract (arXiv)

We introduce EmphAssess, a prosodic benchmark designed to evaluate the capability of speech-to-speech models to encode and reproduce prosodic emphasis. We apply this to two tasks: speech resynthesis and speech-to-speech translation. In both cases, the benchmark evaluates the ability of the model to encode emphasis in the speech input and accurately reproduce it in the output, potentially across a change of speaker and language. As part of the evaluation pipeline, we introduce EmphaClass, a new model that classifies emphasis at the frame or word level.