mlsanityData to 14 Oct 2026

SIFT-50M: A Large-Scale Multilingual Dataset for Speech Instruction Fine-Tuning

arXiv 2504.09081 ↗First posted 12 Apr 2025eess.AS cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We introduce SIFT (Speech Instruction Fine-Tuning), a 50M-example dataset designed for instruction fine-tuning and pre-training of speech-text large language models (LLMs). SIFT-50M is built from publicly available speech corpora, which collectively contain 14K hours of speech, and leverages LLMs along with off-the-shelf expert models. The dataset spans five languages, encompassing a diverse range of speech understanding as well as controllable speech generation instructions. Using SIFT-50M, we train SIFT-LLM, which outperforms existing speech-text LLMs on instruction-following benchmarks while achieving competitive performance on foundational speech tasks. To support further research, we also introduce EvalSIFT, a benchmark dataset specifically designed to evaluate the instruction-following capabilities of speech-text LLMs.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Builds on 36 of its 58 references that are in the map. First posted on arXiv 12 Apr 2025.

Builds on

36 of 58 references.

References · 36

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
  8. 2023Large Language Modelscited by 780
  9. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 762
  10. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  11. 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
  12. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 506
24 more
  1. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  2. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  3. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  4. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  7. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  8. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  9. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
  10. 2024AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehensioncited by 220
  11. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  12. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  13. 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
  14. 2021The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usagecited by 101
  15. 2023Dynamic-SUPERB: Towards A Dynamic, Collaborative, and Comprehensive Instruction-Tuning Benchmark for Speechcited by 101
  16. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  17. 2023LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPTcited by 78
  18. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  19. 2024Dynamic-SUPERB Phase-2: A Collaboratively Expanding Benchmark for Measuring the Capabilities of Spoken Language Models with 180 Taskscited by 71
  20. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  21. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 63
  22. 2023Prompting Large Language Models for Zero-Shot Domain Adaptation in Speech Recognitioncited by 48
  23. 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
  24. 2024LibriTTS-P: A Corpus with Speaking Style and Speaker Identity Prompts for Text-to-Speech and Style Captioningcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 90
  2. 2025DeSTA2.5-Audio: Toward General-Purpose Large Audio Language Model with Self-Generated Cross-Modal Alignmentcited by 50
  3. 2025Towards Holistic Evaluation of Large Audio-Language Models: A Comprehensive Surveycited by 39
  4. 2026Audio Flamingo Next: Next-Generation Open Audio-Language Models for Speech, Sound, and Musiccited by 19
  5. 2025MCIF: Multimodal Crosslingual Instruction-Following Benchmark from Scientific Talkscited by 14
  6. 2025MECAT: A Multi-Experts Constructed Benchmark for Fine-Grained Audio Understanding Taskscited by 6
  7. 2025IFEval-Audio: Benchmarking Instruction-Following Capability in Audio-based Large Language Modelscited by 4
  8. 2025Incorporating Contextual Paralinguistic Understanding in Large Speech-Language Modelscited by 4
  9. 2025EmoSLLM: Parameter-Efficient Adaptation of LLMs for Speech Emotion Recognitioncited by 4
  10. 2025LLaSO: A Foundational Framework for Reproducible Research in Large Language and Speech Modelcited by 3
  11. 2026Adaptive Turn-Taking for Real-time Multi-Party Voice Agentscited by 2
  12. 2026Do What I Say: A Spoken Prompt Dataset for Instruction-Followingcited by 1
2 more
  1. 2026Unlocking Speech-Text Compositional Powers: Instruction-Following Speech Language Models without Instruction Tuningcited by 0
  2. 2026A Factorial Ablation of a Speech-to-SFT Pipeline: Differential Effects on Data Quality and Downstream Transfercited by 0