# Joint Automatic Speech Recognition And Structure Learning For Better Speech Understanding

arXiv: https://arxiv.org/abs/2501.07329 · HTML: https://mlsanity.com/paper/2501.07329/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 29 of its 36 references that are in the map.
- First posted on arXiv 13 Jan 2025.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)
- [SLURP: A Spoken Language Understanding Resource Package](https://mlsanity.com/paper/2011.13205/) (2020)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Semantics-aware BERT for Language Understanding](https://mlsanity.com/paper/1909.02209/) (2019)
- [FunASR: A Fundamental End-to-End Speech Recognition Toolkit](https://mlsanity.com/paper/2305.11013/) (2023)
- [SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speech](https://mlsanity.com/paper/2111.10367/) (2021)
- [Efficient Sequence Transduction by Jointly Predicting Tokens and Durations](https://mlsanity.com/paper/2304.06795/) (2023)
- [SpanNER: Named Entity Re-/Recognition as Span Prediction](https://mlsanity.com/paper/2106.00641/) (2021)
- [ESPnet-SLU: Advancing Spoken Language Understanding through ESPnet](https://mlsanity.com/paper/2111.14706/) (2021)
- [End-to-end Named Entity Recognition from English Speech](https://mlsanity.com/paper/2005.11184/) (2020)
- [BatGPT: A Bidirectional Autoregessive Talker from Generative Pre-trained Transformer](https://mlsanity.com/paper/2307.00360/) (2023)
- [Dependency or Span, End-to-End Uniform Semantic Role Labeling](https://mlsanity.com/paper/1901.05280/) (2019)
- [AISHELL-NER: Named Entity Recognition from Chinese Speech](https://mlsanity.com/paper/2202.08533/) (2022)
- [A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understanding](https://mlsanity.com/paper/2211.05869/) (2022)
- [Global Pointer: Novel Efficient Span-based Approach for Named Entity Recognition](https://mlsanity.com/paper/2208.03054/) (2022)
- [Seeking Common but Distinguishing Difference, A Joint Aspect-based Sentiment Analysis Model](https://mlsanity.com/paper/2111.09634/) (2021)
- [On the Use of External Data for Spoken Named Entity Recognition](https://mlsanity.com/paper/2112.07648/) (2021)
- [Token-level Sequence Labeling for Spoken Language Understanding using Compositional End-to-End Models](https://mlsanity.com/paper/2210.15734/) (2022)
- [UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions](https://mlsanity.com/paper/2310.02973/) (2023)
- [Context-aware Fine-tuning of Self-supervised Speech Models](https://mlsanity.com/paper/2212.08542/) (2022)
- [On joint training with interfaces for spoken language understanding](https://mlsanity.com/paper/2106.15919/) (2021)
- [Sparse Fuzzy Attention for Structured Sentiment Analysis](https://mlsanity.com/paper/2109.06719/) (2021)
- [Introducing Semantics into Speech Encoders](https://mlsanity.com/paper/2211.08402/) (2022)
- [Generative Context-aware Fine-tuning of Self-supervised Speech Models](https://mlsanity.com/paper/2312.09895/) (2023)
- [Leveraging Multilingual Self-Supervised Pretrained Models for Sequence-to-Sequence End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2310.06103/) (2023)

## Abstract (arXiv)

Spoken language understanding (SLU) is a structure prediction task in the field of speech. Recently, many works on SLU that treat it as a sequence-to-sequence task have achieved great success. However, This method is not suitable for simultaneous speech recognition and understanding. In this paper, we propose a joint speech recognition and structure learning framework (JSRSL), an end-to-end SLU model based on span, which can accurately transcribe speech and extract structured content simultaneously. We conduct experiments on name entity recognition and intent classification using the Chinese dataset AISHELL-NER and the English dataset SLURP. The results show that our proposed method not only outperforms the traditional sequence-to-sequence method in both transcription and extraction capabilities but also achieves state-of-the-art performance on the two datasets.