mlsanityData to 7 Oct 2026

Instruction Data Generation and Unsupervised Adaptation for Speech Language Models

arXiv 2406.12946 ↗First posted 18 Jun 2024eess.AS cs.AI cs.CLcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

In this paper, we propose three methods for generating synthetic samples to train and evaluate multimodal large language models capable of processing both text and speech inputs. Addressing the scarcity of samples containing both modalities, synthetic data generation emerges as a crucial strategy to enhance the performance of such systems and facilitate the modeling of cross-modal relationships between the speech and text domains. Our process employs large language models to generate textual components and text-to-speech systems to generate speech components. The proposed methods offer a practical and effective means to expand the training dataset for these models. Experimental results show progress in achieving an integrated understanding of text and speech. We also highlight the potential of using unlabeled speech data to generate synthetic samples comparable in quality to those with available transcriptions, enabling the expansion of these models to more languages.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 27 references that are in the map. First posted on arXiv 18 Jun 2024.

Builds on

19 of 27 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023COSMIC: Data Efficient Instruction-tuning For Speech In-Context Learning3 Nov

Other references · 18

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  3. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  4. 2022Emergent Abilities of Large Language Modelscited by 2,483
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  6. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  7. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  8. 2015Listen, Attend and Spellcited by 542
  9. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  11. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  12. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
6 more
  1. 2020FastPitch: Parallel Text-to-speech with Pitch Predictioncited by 150
  2. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  3. 2020Improved Noisy Student Training for Automatic Speech Recognitioncited by 139
  4. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  5. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  6. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 4.

9 papers.

  1. 2025SALM-Duplex: Efficient and Direct Duplex Modeling for Speech-to-Speech Language Modelcited by 32
  2. 2024BESTOW: Efficient and Streamable Speech Language Model with the Best of Two Worlds in GPT and T5cited by 18
  3. 2025When Large Language Models Meet Speech: A Survey on Integration Approachescited by 9
  4. 2024VoiceTextBlender: Augmenting Large Language Models with Speech Capabilities via Single-Stage Joint Speech-Text Supervised Fine-Tuningcited by 6
  5. 2026The Silent Thought: Modeling Internal Cognition in Full-Duplex Spoken Dialogue Models via Latent Reasoningcited by 5
  6. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  7. 2026MENASpeechBank: A Reference Voice Bank with Persona-Conditioned Multi-Turn Conversations for AudioLLMscited by 3
  8. 2026Language-Aware Distillation for Multilingual Instruction-Following Speech LLMs with ASR-Only Supervisioncited by 1
  9. 2026Aslema at NADI 2026: Data Augmentation for Intent Recognition and Slot Fillingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  2. 2024A Survey on Data Synthesis and Augmentation for Large Language Models
  3. 2024Evaluating Language Models as Synthetic Data Generators
  4. 2025Synthetic Data Generation for Phrase Break Prediction with Large Language Model
  5. 2025Instituto de Telecomunicacc~oes at IWSLT 2025: Aligning Small-Scale Speech and Language Models for Speech-to-Text Learning
  6. 2025From Alignment to Advancement: Bootstrapping Audio-Language Alignment with Synthetic Data
  7. 2023Text Generation with Speech Synthesis for ASR Data Augmentation
  8. 2025Modular Techniques for Synthetic Long-Context Data Generation in Language Model Training and Evaluation