mlsanityData to 13 Oct 2026

ASR-Generated Text for Language Model Pre-training Applied to Speech Tasks

arXiv 2207.01893 ↗First posted 5 Jul 2022cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

We aim at improving spoken language modeling (LM) using very large amount of automatically transcribed speech. We leverage the INA (French National Audiovisual Institute) collection and obtain 19GB of text after applying ASR on 350,000 hours of diverse TV shows. From this, spoken language models are trained either by fine-tuning an existing LM (FlauBERT) or through training a LM from scratch. New models (FlauBERT-Oral) are shared with the community and evaluated for 3 downstream tasks: spoken language understanding, classification of TV shows and speech syntactic parsing. Results show that FlauBERT-Oral can be beneficial compared to its initial FlauBERT version demonstrating that, despite its inherent noisy nature, ASR-generated text can be used to build spoken language models.

Key facts. Cited by 6 papers in the map (data through 13 Oct 2026). Builds on 18 of its 36 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

18 of 36 references.

References · 18

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,870
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,313
  5. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,366
  6. 2019CamemBERT: a Tasty French Language Modelcited by 368
  7. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  8. 2021BERT: A Review of Applications in Natural Language Processing and Understandingcited by 109
  9. 2018Towards end-to-end spoken language understandingcited by 92
  10. 2021A Survey on Spoken Language Understanding: Recent Advances and New Frontierscited by 53
  11. 2020Jointly Fine-Tuning "BERT-like" Self Supervised Models to Improve Multimodal Speech Emotion Recognitioncited by 38
  12. 2021Where are we in semantic concept extraction for Spoken Language Understanding?cited by 11
6 more
  1. 2019BERTphone: Phonetically-Aware Encoder Representations for Utterance-Level Speaker and Language Recognitioncited by 10
  2. 2019A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systemscited by 10
  3. 2021End2End Acoustic to Semantic Transductioncited by 10
  4. 2020A Data Efficient End-To-End Spoken Language Understanding Architecturecited by 8
  5. 2020Jointly Encoding Word Confusion Network and Dialogue Context with BERT for Spoken Language Understandingcited by 8
  6. 2021What BERT Based Language Models Learn in Spoken Transcripts: An Empirical Studycited by 5

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 0; 2026: 2.

6 papers.

  1. 2023Modeling Spoken Information Queries for Virtual Assistants: Open Problems, Challenges and Opportunitiescited by 4
  2. 2023Server-side Rescoring of Spoken Entity-centric Knowledge Queries for Virtual Assistantscited by 2
  3. 2024New Semantic Task for the French Spoken Language Understanding MEDIA Benchmarkcited by 2
  4. 2024Automatic Classification of News Subjects in Broadcast News: Application to a Gender Bias Representation Analysiscited by 2
  5. 2026Pantagruel: Unified Self-Supervised Encoders for French Text and Speechcited by 2
  6. 2026RO-N3WS: Enhancing Generalization in Low-Resource ASR with Diverse Romanian Speech Benchmarkscited by 1