mlsanityData to 14 Oct 2026

ASTRA: Aligning Speech and Text Representations for Asr without Sampling

arXiv 2406.06664 ↗First posted 10 Jun 2024eess.AS cs.LG cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper introduces ASTRA, a novel method for improving Automatic Speech Recognition (ASR) through text injection.Unlike prevailing techniques, ASTRA eliminates the need for sampling to match sequence lengths between speech and text modalities. Instead, it leverages the inherent alignments learned within CTC/RNNT models. This approach offers the following two advantages, namely, avoiding potential misalignment between speech and text features that could arise from upsampling and eliminating the need for models to accurately predict duration of sub-word tokens. This novel formulation of modality (length) matching as a weighted RNNT objective matches the performance of the state-of-the-art duration-based methods on the FLEURS benchmark, while opening up other avenues of research in speech processing.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 25 of its 30 references that are in the map. First posted on arXiv 10 Jun 2024.

Builds on

25 of 30 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2022FLEURS: Few-shot Learning Evaluation of Universal Representations of Speechcited by 469
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  9. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  10. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  11. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  12. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
13 more
  1. 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
  2. 2020Parallel Tacotron: Non-Autoregressive and Controllable TTScited by 49
  3. 2020Alignment Restricted Streaming Recurrent Neural Network Transducercited by 42
  4. 2023Pre-training for Speech Translation: CTC Meets Optimal Transportcited by 25
  5. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  6. 2022Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systemscited by 20
  7. 2022Virtuoso: Massive Multilingual Speech-Text Joint Semi-Supervised Learning for Text-To-Speechcited by 15
  8. 2022Maestro-U: Leveraging joint speech-text representation learning for zero supervised speech ASRcited by 13
  9. 2023Decoder-only Architecture for Speech Recognition with CTC Prompts and Text Data Augmentationcited by 10
  10. 2023Understanding Shared Speech-Text Representationscited by 6
  11. 2023Improving Joint Speech-Text Representations Without Alignmentcited by 4
  12. 2023USM-SCD: Multilingual Speaker Change Detection Based on Large Pretrained Foundation Modelscited by 4
  13. 2023Using Text Injection to Improve Recognition of Personal Identifiers in Speechcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Speechless: Speech Instruction Training Without Speech for Low Resource Languagescited by 4
  2. 2026Text-Utilization for Encoder-dominated Speech Recognition Modelscited by 0