mlsanityData to 14 Oct 2026

Improving Joint Speech-Text Representations Without Alignment

arXiv 2308.06125 ↗First posted 11 Aug 2023cs.CL cs.AI cs.SDcited by 4

Not on the map. Why?

Abstract (arXiv)

The last year has seen astonishing progress in text-prompted image generation premised on the idea of a cross-modal representation space in which the text and image domains are represented jointly. In ASR, this idea has found application as joint speech-text encoders that can scale to the capacities of very large parameter models by being trained on both unpaired speech and text. While these methods show promise, they have required special treatment of the sequence-length mismatch inherent in speech and text, either by up-sampling heuristics or an explicit alignment model. In this work, we offer evidence that joint speech-text encoders naturally achieve consistent representations across modalities by disregarding sequence length, and argue that consistency losses could forgive length differences and simply assume the best alignment. We show that such a loss improves downstream WER in both a large-parameter monolingual and multilingual system.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 18 of its 20 references that are in the map. First posted on arXiv 11 Aug 2023.

Builds on

18 of 20 references.

References · 18

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  8. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  9. 2020Hybrid Autoregressive Transducer (hat)cited by 108
  10. 2022mSLAM: Massively multilingual joint pre-training for speech and textcited by 108
  11. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  12. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
6 more
  1. 2020Leveraging Unpaired Text Data for Training End-to-End Speech-to-Intent Systemscited by 38
  2. 2021Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencodercited by 27
  3. 2022JOIST: A Joint Speech and Text Streaming Model For ASRcited by 24
  4. 2022Towards Reducing the Need for Speech Training Data To Build Spoken Language Understanding Systemscited by 20
  5. 2018Multi-Modal Data Augmentation for End-to-End ASRcited by 18
  6. 2022Learning the joint distribution of two sequences using little or no paired datacited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 0; 2026: 1.

4 papers.

  1. 2024Text Injection for Neural Contextual Biasingcited by 3
  2. 2024Efficiently Train ASR Models that Memorize Less and Perform Better with Per-core Clippingcited by 2
  3. 2024ASTRA: Aligning Speech and Text Representations for Asr without Samplingcited by 2
  4. 2026Rethinking Speech-LLM Integration for ASR: Effective Joint Speech-Text Training by Interleavingcited by 0