# PACT: A Pragmatic Prompting Framework for Expressive and Voice-Preserving Speech-to-Speech Translation

arXiv: https://arxiv.org/abs/2610.18247 · HTML: https://mlsanity.com/paper/2610.18247/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 25 of its 32 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Builds on (most cited first, 25 of 25)

- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokens](https://mlsanity.com/paper/2407.05407/) (2024)
- [emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation](https://mlsanity.com/paper/2312.15185/) (2023)
- [Seamless: Multilingual Expressive and Streaming Speech Translation](https://mlsanity.com/paper/2312.05187/) (2023)
- [XTTS: a Massively Multilingual Zero-Shot Text-to-Speech Model](https://mlsanity.com/paper/2406.04904/) (2024)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [Direct speech-to-speech translation with discrete units](https://mlsanity.com/paper/2107.05604/) (2021)
- [Translatotron 2: High-quality direct speech-to-speech translation with voice preservation](https://mlsanity.com/paper/2107.08661/) (2021)
- [Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis](https://mlsanity.com/paper/2411.01156/) (2024)
- [Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation](https://mlsanity.com/paper/2204.02967/) (2022)
- [OpenVoice: Versatile Instant Voice Cloning](https://mlsanity.com/paper/2312.01479/) (2023)
- [A Holistic Cascade System, benchmark, and Human Evaluation Protocol for Expressive Speech-to-Speech Translation](https://mlsanity.com/paper/2301.10606/) (2023)
- [UniSS: Unified Expressive Speech-to-Speech Translation with Your Voice](https://mlsanity.com/paper/2509.21144/) (2025)
- [Speech-to-Speech Translation with Discrete-Unit-Based Style Transfer](https://mlsanity.com/paper/2309.07566/) (2023)
- [Direct Speech-to-Speech Neural Machine Translation: A Survey](https://mlsanity.com/paper/2411.14453/) (2024)
- [Dialogs Re-enacted Across Languages](https://mlsanity.com/paper/2211.11584/) (2022)
- [Expressive Machine Dubbing Through Phrase-level Cross-lingual Prosody Transfer](https://mlsanity.com/paper/2306.11662/) (2023)
- [CrossVoice: Crosslingual Prosody Preserving Cascade-S2ST using Transfer Learning](https://mlsanity.com/paper/2406.00021/) (2024)
- [Representing 'how you say' with 'what you say': English corpus of focused speech and text reflecting corresponding implications](https://mlsanity.com/paper/2203.15483/) (2022)
- [A Unit-based System and Dataset for Expressive Direct Speech-to-Speech Translation](https://mlsanity.com/paper/2502.00374/) (2025)
- [Dub-S2ST: Textless Speech-to-Speech Translation for Seamless Dubbing](https://mlsanity.com/paper/2505.20899/) (2025)

## Abstract (arXiv)

Speech-to-speech translation should preserve not only what is said, but also who says it and how it is said. Existing end-to-end S2ST, speech MLLM, and cascade systems can preserve content or timbre, but often struggle to realize emotion, tone, rhythm, pauses, and pragmatic attitude in the target language. We propose PACT, a target-space pragmatic prompting framework for expressive and voice-preserving S2ST. PACT extracts source pragmatic cues, projects them into the target-language expression space, calibrates the projected representation to a realizable target subspace, and converts it into continuous prompts for a frozen speech generator. On bidirectional Spanish-English TOPI evaluation, PACT achieves the best overall performance among evaluated systems across content fidelity, voice preservation, and expression preservation metrics.