mlsanityData to 14 Oct 2026

GRASS: Unified Generation Model for Speech-to-Semantic Tasks

arXiv 2309.02780 ↗First posted 6 Sep 2023cs.CL cs.SD eess.AScited by 0

Not on the map. Why?

Abstract (arXiv)

This paper explores the instruction fine-tuning technique for speech-to-semantic tasks by introducing a unified end-to-end (E2E) framework that generates target text conditioned on a task-related prompt for audio data. We pre-train the model using large and diverse data, where instruction-speech pairs are constructed via a text-to-speech (TTS) system. Extensive experiments demonstrate that our proposed model achieves state-of-the-art (SOTA) results on many benchmarks covering speech named entity recognition, speech sentiment analysis, speech question answering, and more, after fine-tuning. Furthermore, the proposed model achieves competitive performance in zero-shot and few-shot scenarios. To facilitate future work on instruction fine-tuning for speech-to-semantic tasks, we release our instruction dataset and code.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 21 references that are in the map. First posted on arXiv 6 Sep 2023.

Builds on

18 of 21 references.

References · 18

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  3. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  6. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  7. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  8. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  9. 2021Unsupervised Speech Recognitioncited by 148
  10. 2018Towards end-to-end spoken language understandingcited by 92
  11. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 72
  12. 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
6 more
  1. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  2. 2022SLUE Phase-2: A Benchmark Suite of Diverse Spoken Language Understanding Taskscited by 35
  3. 2021Speech-language Pre-training for End-to-end Spoken Language Understandingcited by 27
  4. 2019Using Speech Synthesis to Train End-to-End Spoken Language Understanding Modelscited by 24
  5. 2021On the Use of External Data for Spoken Named Entity Recognitioncited by 12
  6. 2022End-to-end model for named entity recognition from speech without paired training datacited by 11

Built on by

Not cited yet.