mlsanityData to 14 Oct 2026

Understanding Shared Speech-Text Representations

arXiv 2304.14514 ↗First posted 27 Apr 2023cs.CL cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

Recently, a number of approaches to train speech models by incorpo-rating text into end-to-end models have been developed, with Mae-stro advancing state-of-the-art automatic speech recognition (ASR)and Speech Translation (ST) performance. In this paper, we expandour understanding of the resulting shared speech-text representationswith two types of analyses. First we examine the limits of speech-free domain adaptation, finding that a corpus-specific duration modelfor speech-text alignment is the most important component for learn-ing a shared speech-text representation. Second, we inspect the sim-ilarities between activations of unimodal (speech or text) encodersas compared to the activations of a shared encoder. We find that theshared encoder learns a more compact and overlapping speech-textrepresentation than the uni-modal encoders. We hypothesize that thispartially explains the effectiveness of the Maestro shared speech-textrepresentations.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 22 of its 29 references that are in the map. First posted on arXiv 27 Apr 2023.

Builds on

22 of 29 references.

References · 22

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  3. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  4. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  5. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  6. 2021Unsupervised Speech Recognitioncited by 148
  7. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  8. 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
  9. 2019Learning Relationships between Text, Audio, and Video via Deep Canonical Correlation for Multimodal Language Analysiscited by 82
  10. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  11. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  12. 2021Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translationcited by 47
10 more
  1. 2022ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Trainingcited by 47
  2. 2015Deep convolutional acoustic word embeddings using word-pair side informationcited by 27
  3. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  4. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  5. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  6. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  7. 2022Modular Hybrid Autoregressive Transducercited by 17
  8. 2020Multilingual Jointly Trained Acoustic and Written Word Embeddingscited by 16
  9. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  10. 2022Tokenwise Contrastive Pretraining for Finer Speech-to-BERT Alignment in End-to-End Speech-to-Intent Systemscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 3; 2025: 0; 2026: 1.

6 papers.

  1. 2023Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representationcited by 4
  2. 2023Using Text Injection to Improve Recognition of Personal Identifiers in Speechcited by 3
  3. 2024ASTRA: Aligning Speech and Text Representations for Asr without Samplingcited by 2
  4. 2024UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Modelscited by 1
  5. 2024High-precision Voice Search Query Correction via Retrievable Speech-text Embedingscited by 0
  6. 2026Text-Utilization for Encoder-dominated Speech Recognition Modelscited by 0