mlsanityData to 7 Oct 2026

Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs

arXiv 2503.06362 ↗First posted 9 Mar 2025cs.CV cs.MM cs.SDcited by 11

On the map: Multimodal LLMs / vision-language models → Video understanding with MLLMs

Abstract (arXiv)

Audio-Visual Speech Recognition (AVSR) leverages audio and visual modalities to improve robustness in noisy environments. Recent advances in Large Language Models (LLMs) show strong performance in speech recognition, including AVSR. However, the long speech representations lead to high computational costs for LLMs. Prior methods compress inputs before feeding them to LLMs, but high compression often harms accuracy. To address this, we propose Llama-MTSK, the first Matryoshka-based Multimodal LLM for AVSR, which flexibly adapts audio-visual token allocation under varying compute constraints. Inspired by Matryoshka Representation Learning, our model encodes representations at multiple granularities with a single architecture, avoiding the need for separate models. For efficient fine-tuning, we introduce three LoRA-based strategies using global and scale-specific modules. Evaluations on major AVSR datasets show Llama-MTSK matches or outperforms models trained at fixed compression levels.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Video understanding with MLLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 64 of its 73 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

64 of 73 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Large Language Models are Strong Audio-Visual Speech Recognition Learners18 Sep

Other references · 63

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  9. 2024Mixtral of Expertscited by 1,785
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  12. 2023VILA: On Pre-training for Visual Language Modelscited by 686
38 more
  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  2. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2024LoRA+: Efficient Low Rank Adaptation of Large Modelscited by 365
  5. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  6. 2018Deep Audio-Visual Speech Recognitioncited by 318
  7. 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
  8. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  9. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  10. 2022Matryoshka Representation Learningcited by 268
  11. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  12. 2023From Sparse to Soft Mixtures of Expertscited by 245
  13. 2016Lip Reading Sentences in the Wildcited by 226
  14. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  15. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  16. 2023Honeybee: Locality-enhanced Projector for Multimodal LLMcited by 195
  17. 2024LLaMA-Omni: Seamless Speech Interaction with Large Language Modelscited by 169
  18. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  19. 2021End-to-end Audio-visual Speech Recognition with Conformerscited by 147
  20. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 139
  21. 2023Auto-AVSR: Audio-Visual Speech Recognition with Automatic Labelscited by 113
  22. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  23. 2024Multimodal Autoregressive Pre-training of Large Vision Encoderscited by 108
  24. 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 86
  25. 2024MoVA: Adapting Mixture of Vision Experts to Multimodal Contextcited by 82
  26. 2024Matryoshka Multimodal Modelscited by 82
  27. 2023Connecting Speech Encoder and Large Language Model for ASRcited by 79
  28. 2023MatFormer: Nested Transformer for Elastic Inferencecited by 79
  29. 2018Audio-Visual Speech Recognition With A Hybrid CTC/Attention Architecturecited by 61
  30. 2024CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Expertscited by 56
  31. 2024Matryoshka Query Transformer for Large Vision-Language Modelscited by 49
  32. 2024Dense Connector for MLLMscited by 48
  33. 2022Jointly Learning Visual and Auditory Speech Representations from Raw Datacited by 43
  34. 2024DeSTA2: Developing Instruction-Following Speech Language Model Without Speech Instruction-Tuning Datacited by 43
  35. 2023Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoringcited by 42
  36. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 40
  37. 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
  38. 2024Whisper-Flamingo: Integrating Visual Features into Whisper for Audio-Visual Speech Recognition and Translationcited by 35

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  2. 2025Scaling and Enhancing LLM-based AVSR: A Sparse Mixture of Projectors Approachcited by 10
  3. 2025Mitigating Attention Sinks and Massive Activations in Audio-Visual Speech Recognition with LLMscited by 9
  4. 2025MoME: Mixture of Matryoshka Experts for Audio-Visual Speech Recognitioncited by 4
  5. 2025Omni-AVSR: Towards Unified Multimodal Speech Recognition with Large Language Modelscited by 3
  6. 2025MLLM-based Speech Recognition: When and How is Multimodality Beneficial?cited by 1
  7. 2025Two Heads Are Better Than One: Audio-Visual Speech Error Correction with Dual Hypothesescited by 1
  8. 2026Dr. SHAP-AV: Decoding Relative Modality Contributions via Shapley Attribution in Audio-Visual Speech Recognitioncited by 1
  9. 2026Cross-lingual Matryoshka Representation Learning across Speech and Textcited by 0
  10. 2026VIB-AVSR: Variational Information Bottleneck for Noise-Robust LLM-Based Audio-Visual Speech Recognitioncited by 0
  11. 2026Attention-Guided Reliability Scaling for Contrastive Decoding in Robust Audio-Visual Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoHAVE: Mixture of Hierarchical Audio-Visual Experts for Robust Speech Recognition
  2. 2025AccKV: Towards Efficient Audio-Video LLMs Inference via Adaptive-Focusing and Cross-Calibration KV Cache Optimization
  3. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Models
  4. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Models
  5. 2023Audio-Visual LLM for Video Understanding
  6. 2025Aligned Better, Listen Better for Audio-Visual Large Language Models
  7. 2024Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time
  8. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding