CIF-PT: Bridging Speech and Text Representations for Spoken Language Understanding via Continuous Integrate-and-Fire Pre-Training
Not on the map. Why?
Abstract (arXiv)
Speech or text representation generated by pre-trained models contains modal-specific information that could be combined for benefiting spoken language understanding (SLU) tasks. In this work, we propose a novel pre-training paradigm termed Continuous Integrate-and-Fire Pre-Training (CIF-PT). It relies on a simple but effective frame-to-token alignment: continuous integrate-and-fire (CIF) to bridge the representations between speech and text. It jointly performs speech-to-text training and language model distillation through CIF as the pre-training (PT). Evaluated on SLU benchmark SLURP dataset, CIF-PT outperforms the state-of-the-art model by 1.94% of accuracy and 2.71% of SLU-F1 on the tasks of intent classification and slot filling, respectively. We also observe the cross-modal representation extracted by CIF-PT obtains better performance than other neural interfaces for the tasks of SLU, including the dominant speech representation learned from self-supervised pre-training.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 27 of its 41 references that are in the map. First posted on arXiv 27 May 2023.
Builds on
27 of 41 references.
References · 27
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2015Listen, Attend and Spellcited by 542
- 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
- 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
15 more
- 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
- 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
- 2020fairseq S2T: Fast Speech-to-Text Modeling with fairseqcited by 174
- 2018Towards end-to-end spoken language understandingcited by 92
- 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
- 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
- 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
- 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
- 2018From Audio to Semantics: Approaches to end-to-end spoken language understandingcited by 63
- 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
- 2019Self-Attention Aligner: A Latency-Control End-to-End Model for ASR Using Self-Attention Network and Chunk-Hoppingcited by 33
- 2020Speech To Semantics: Improve ASR and NLU Jointly via All-Neural Interfacescited by 24
- 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understandingcited by 22
- 2020Two-stage Textual Knowledge Distillation for End-to-End Spoken Language Understandingcited by 9
- 2021On joint training with interfaces for spoken language understandingcited by 5
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.