mlsanityData to 14 Oct 2026

SSR: Alignment-Aware Modality Connector for Speech Language Models

arXiv 2410.00168 ↗First posted 30 Sep 2024cs.CL cs.SD eess.AScited by 16

Not on the map. Why?

Abstract (arXiv)

Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-Connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-Connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., +10 accuracy on StoryCloze and +20 on Speech-MMLU) while preserving pre-trained text ability.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 45 of its 62 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

45 of 62 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
33 more
  1. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  2. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 616
  4. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  5. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  6. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  7. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  8. 2023VideoPoet: A Large Language Model for Zero-Shot Video Generationcited by 492
  9. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  10. 2023Scaling Speech Technology to 1,000+ Languagescited by 411
  11. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  12. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  13. 2021Generative Spoken Language Modeling from Raw Audiocited by 340
  14. 2020Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Searchcited by 273
  15. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 265
  16. 2023Seamless: Multilingual Expressive and Streaming Speech Translationcited by 192
  17. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 172
  18. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 145
  19. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  20. 2024Spirit LM: Interleaved Spoken and Written Language Modelcited by 126
  21. 2020The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modelingcited by 101
  22. 2023Textually Pretrained Speech Language Modelscited by 101
  23. 2023EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesiscited by 98
  24. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 80
  25. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  26. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 72
  27. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  28. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 37
  29. 2022Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systemscited by 28
  30. 2021Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systemscited by 15
  31. 2023ZMM-TTS: Zero-shot Multilingual and Multispeaker Speech Synthesis Conditioned on Self-supervised Discrete Speech Representationscited by 15
  32. 2024Less Peaky and More Accurate CTC Forced Alignment by Label Priorscited by 12
  33. 2024Streaming Sequence Transduction through Dynamic Compressioncited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 7.

16 papers.

Sort:

By area:

  1. 2025Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMscited by 11
  2. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 10
  3. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  4. 2025Can LLMs Understand Unvoiced Speech? Exploring EMG-to-Text Conversion with LLMscited by 6
  5. 2025LegoSLM: Connecting LLM with Speech Encoder using CTC Posteriorscited by 4
  6. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  7. 2026SpeechMapper: Speech-to-text Embedding Projector for LLMscited by 4
  8. 2026Beyond Fixed Frames: Dynamic Character-Aligned Speech Tokenizationcited by 4
  9. 2026Bias in the Ear of the Listener: Assessing Sensitivity in Audio Language Models Across Linguistic, Demographic, and Positional Variationscited by 4
  10. 2025ALAS: An Automatic Latent Alignment Score for Audio Language Modelscited by 3
  11. 2025Language Models as Semantic Teachers: Post-Training Alignment for Medical Audio Understandingcited by 1
  12. 2026Is Text All You Need? Text as a Universal Information Bottleneck for Speech LLMscited by 1
4 more
  1. 2025Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generationcited by 0
  2. 2026Speech2Grasp: Data-Efficient Transfer of Text-Conditioned Grasp Detection to Speech in Humanoid Robotscited by 0
  3. 2026Align, Integrate, and Fire: Efficient Token-Level Alignment for Zero-Shot SpeechLLMscited by 0
  4. 2026Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Understanding the Modality Gap: An Empirical Study on the Speech-Text Alignment Mechanism of Large Speech Language Models
  2. 2022An Analysis of Semantically-Aligned Speech-Text Embeddings
  3. 2025Adaptive Inner Speech-Text Alignment for LLM-based Speech Translation
  4. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  5. 2025Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment
  6. 2026Closing the Modality Reasoning Gap for Speech Large Language Models
  7. 2024DeSTA: Enhancing Speech Language Models through Descriptive Speech-Text Alignment
  8. 2026Gradient-Based Speech-to-Text Alignment for Any ASR Model: From CTC to Speech LLMs