mlsanityData to 7 Oct 2026

SynesLM: A Unified Approach for Audio-visual Speech Recognition and Translation via Language Model and Synthetic Data

arXiv 2408.00624 ↗First posted 1 Aug 2024eess.AS cs.CL cs.CVcited by 3

Not on the map. Why?

Abstract (arXiv)

In this work, we present SynesLM, an unified model which can perform three multimodal language understanding tasks: audio-visual automatic speech recognition(AV-ASR) and visual-aided speech/machine translation(VST/VMT). Unlike previous research that focused on lip motion as visual cues for speech signals, our work explores more general visual information within entire frames, such as objects and actions. Additionally, we use synthetic image data to enhance the correlation between image and speech data. We benchmark SynesLM against the How2 dataset, demonstrating performance on par with state-of-the-art (SOTA) models dedicated to AV-ASR while maintaining our multitasking framework. Remarkably, for zero-shot AV-ASR, SynesLM achieved SOTA performance by lowering the Word Error Rate (WER) from 43.4% to 39.4% on the VisSpeech Dataset. Furthermore, our results in VST and VMT outperform the previous results, improving the BLEU score to 43.5 from 37.2 for VST, and to 54.8 from 54.4 for VMT.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 42 of its 44 references that are in the map. First posted on arXiv 1 Aug 2024.

Builds on

42 of 44 references.

References · 42

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  10. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  11. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  12. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
30 more
  1. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  2. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  3. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 786
  4. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  5. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  6. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  8. 2023SALMONN: Towards Generic Hearing Abilities for Large Language Modelscited by 492
  9. 2021Generative Spoken Language Modeling from Raw Audiocited by 336
  10. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 333
  11. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
  12. 2023OneLLM: One Framework to Align All Modalities with Languagecited by 185
  13. 2023Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and Text Integrationcited by 168
  14. 2023On decoder-only architecture for speech-to-text and large language model integrationcited by 144
  15. 2018How2: A Large-scale Dataset for Multimodal Language Understandingcited by 137
  16. 2023X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languagescited by 127
  17. 2024video-SALMONN: Speech-Enhanced Audio-Visual Large Language Modelscited by 101
  18. 2023Textually Pretrained Speech Language Modelscited by 99
  19. 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
  20. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 75
  21. 2023Voxtlm: unified decoder-only models for consolidating speech recognition/synthesis and speech/text continuation taskscited by 71
  22. 2023Prompting the Hidden Talent of Web-Scale Speech Models for Zero-Shot Task Generalizationcited by 43
  23. 2023Exploring Speech Recognition, Translation, and Understanding with Discrete Speech Units: A Comparative Studycited by 42
  24. 2023PolyVoice: Language Models for Speech to Speech Translationcited by 19
  25. 2023AVFormer: Injecting Vision into Frozen Speech Models for Zero-Shot AV-ASRcited by 16
  26. 2022AVATAR: Unconstrained Audiovisual Speech Recognitioncited by 12
  27. 2020Multiresolution and Multimodal Speech Recognition with Transformerscited by 10
  28. 2020Listen, Look and Deliberate: Visual context-aware speech recognition using pre-trained text-video representationscited by 10
  29. 2018Multimodal Grounding for Sequence-to-Sequence Speech Recognitioncited by 8
  30. 2024TMT: Tri-Modal Translation between Speech, Image, and Text by Processing Different Modalities as Different Languagescited by 4

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 1.

3 papers.

  1. 2024FastAdaSP: Multitask-Adapted Efficient Inference for Large Speech Language Modelcited by 5
  2. 2025ViDove: A Translation Agent System with Multimodal Context and Memory-Augmented Reasoningcited by 3
  3. 2024Enhancing Audiovisual Speech Recognition through Bifocal Preference Optimizationcited by 1