# Speechworthy Instruction-tuned Language Models

arXiv: https://arxiv.org/abs/2409.14672 · HTML: https://mlsanity.com/paper/2409.14672/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 66 references that are in the map.
- First posted on arXiv 23 Sep 2024.

## Builds on (most cited first, 30 of 37)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [SimPO: Simple Preference Optimization with a Reference-Free Reward](https://mlsanity.com/paper/2405.14734/) (2024)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale](https://mlsanity.com/paper/2306.15687/) (2023)
- [Understanding Dataset Difficulty with V-Usable Information](https://mlsanity.com/paper/2110.08420/) (2021)
- [A Long Way to Go: Investigating Length Correlations in RLHF](https://mlsanity.com/paper/2310.03716/) (2023)
- [AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head](https://mlsanity.com/paper/2304.12995/) (2023)
- [Bias Runs Deep: Implicit Reasoning Biases in Persona-Assigned LLMs](https://mlsanity.com/paper/2311.04892/) (2023)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [fairseq S2T: Fast Speech-to-Text Modeling with fairseq](https://mlsanity.com/paper/2010.05171/) (2020)
- [Poly-encoders: Transformer Architectures and Pre-training Strategies for Fast and Accurate Multi-sentence Scoring](https://mlsanity.com/paper/1905.01969/) (2019)
- [NaturalSpeech: End-to-End Text to Speech Synthesis with Human-Level Quality](https://mlsanity.com/paper/2205.04421/) (2022)
- [Dialogue Response Ranking Training with Large-Scale Human Feedback Data](https://mlsanity.com/paper/2009.06978/) (2020)
- [Coverage-based Example Selection for In-Context Learning](https://mlsanity.com/paper/2305.14907/) (2023)

## Built on by (most cited first, 7 of 7)

- [A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications](https://mlsanity.com/paper/2410.15595/) (2024)
- [DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues](https://mlsanity.com/paper/2607.26178/) (2026)
- [Toward Machine Interpreting: Lessons from Human Interpreting Studies](https://mlsanity.com/paper/2508.07964/) (2025)
- [Same Words, Different Judgments: How Preferences Vary Across Modalities](https://mlsanity.com/paper/2602.22710/) (2026)
- [Speech-Worthy Alignment for Japanese SpeechLLMs via Direct Preference Optimization](https://mlsanity.com/paper/2603.12565/) (2026)
- [Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation](https://mlsanity.com/paper/2605.27136/) (2026)
- [Ready to Speak: Aligning LLMs for TTS-Friendly Text Generation](https://mlsanity.com/paper/2609.01246/) (2026)

## Abstract (arXiv)

Current instruction-tuned language models are exclusively trained with textual preference data and thus are often not aligned with the unique requirements of other modalities, such as speech. To better align language models with the speech domain, we explore (i) prompting strategies grounded in radio-industry best practices and (ii) preference learning using a novel speech-based preference data of 20K samples, generated with a wide spectrum of prompts that induce varying dimensions of speech-suitability and labeled by annotators who listen to response pairs. Both human and automatic evaluation show that both prompting and preference learning increase the speech-suitability of popular instruction-tuned LLMs. Interestingly, we find that prompting and preference learning can be additive; combining them achieves the best win rates in head-to-head comparison, resulting in responses that are preferred or tied to the base model in 76.2% of comparisons on average. Lastly, we share lexical, syntactical, and qualitative analyses to showcase how each method contributes to improving the speech-suitability of generated responses.