mlsanityData to 14 Oct 2026

Multimodal Large Language Model-Enabled Video Translation: A Role-Oriented Survey

arXiv 2604.11283 ↗First posted 13 Apr 2026cs.CVcited by 0

On the map: LLM reasoning → Audio-language model reasoning

also in 1 more
Abstract (arXiv)

Recent progress in multimodal large language models (MLLMs) is reshaping video translation from a cascaded pipeline of automatic speech recognition, machine translation, text-to-speech, and lip synchronization into a unified multimodal reasoning and generation problem. High-quality video translation requires not only semantic fidelity, but also temporal alignment, speaker consistency, and emotional expressiveness across visual, acoustic, and linguistic streams. This survey provides a focused review of MLLM-enabled video translation through a role-oriented taxonomy. We organize MLLM-enabled and MLLM-relevant studies into three functional roles: Semantic Reasoner, which grounds translation in video understanding, temporal reasoning, and multimodal fusion; Expressive Performer, which supports controllable and context-aware speech generation; and Visual Synthesizer, which enables lip synchronization and visually coherent speaker rendering. We further summarize representative datasets, benchmarks, and metrics for each role, and discuss how current evaluation protocols fall short of end-to-end video translation requirements. Finally, we identify open challenges in long-form video understanding, temporal modeling, multimodal alignment, multilingual robustness, and responsible deployment, outlining future directions for natural and trustworthy cross-lingual video communication.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Audio-language model reasoning, one of 17 directions in LLM reasoning. Builds on 64 of its 176 references that are in the map. First posted on arXiv 13 Apr 2026.

Builds on

64 of 176 references.

References · 64

  1. 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
  2. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  3. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  5. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  6. 2024HunyuanVideo: A Systematic Framework For Large Video Generative Modelscited by 1,660
  7. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,433
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2022CogVideo: Large-scale Pretraining for Text-to-Video Generation via Transformerscited by 985
  10. 2023VideoChat: Chat-Centric Video Understandingcited by 857
  11. 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
  12. 2023VideoCrafter1: Open Diffusion Models for High-Quality Video Generationcited by 535
38 more
  1. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 518
  2. 2020MLS: A Large-Scale Multilingual Dataset for Speech Researchcited by 476
  3. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  4. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  5. 2024CosyVoice: A Scalable Multilingual Zero-shot Text-to-speech Synthesizer based on Supervised Semantic Tokenscited by 334
  6. 2024PLLaVA : Parameter-free LLaVA Extension from Images to Videos for Video Dense Captioningcited by 322
  7. 2024Seed-TTS: A Family of High-Quality Versatile Speech Generation Modelscited by 300
  8. 2024CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Modelscited by 288
  9. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  10. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  11. 2023Valley: Video Assistant with Large Language model Enhanced abilitYcited by 234
  12. 2024NaturalSpeech 3: Zero-Shot Speech Synthesis with Factorized Codec and Diffusion Modelscited by 226
  13. 2024MaskGCT: Zero-Shot Text-to-Speech with Masked Generative Codec Transformercited by 161
  14. 2024Vidu: a Highly Consistent, Dynamic and Skilled Text-to-Video Generator with Diffusion Modelscited by 160
  15. 2024E2 TTS: Embarrassingly Easy Fully Non-Autoregressive Zero-Shot TTScited by 152
  16. 2025Time-R1: Post-Training Large Vision Language Model for Temporal Video Groundingcited by 146
  17. 2025OmniHuman-1: Rethinking the Scaling-Up of One-Stage Conditioned Human Animation Modelscited by 129
  18. 2025Phantom: Subject-consistent video generation via cross-modal alignmentcited by 129
  19. 2023SeamlessM4T: Massively Multilingual & Multimodal Machine Translationcited by 125
  20. 2024Momentor: Advancing Video Large Language Model with Fine-Grained Temporal Reasoningcited by 122
  21. 2024MiniGPT4-Video: Advancing Multimodal LLMs for Video Understanding with Interleaved Visual-Textual Tokenscited by 118
  22. 2025Spark-TTS: An Efficient LLM-Based Text-to-Speech Model with Single-Stream Decoupled Speech Tokenscited by 113
  23. 2023InstructTTS: Modelling Expressive TTS in Discrete Latent Space with Natural Language Style Promptcited by 92
  24. 2024LatentSync: Taming Audio-Conditioned Latent Diffusion Models for Lip Sync with SyncNet Supervisioncited by 56
  25. 2023PG-Video-LLaVA: Pixel Grounding Large Video-Language Modelscited by 51
  26. 2023PromptTTS 2: Describing and Generating Voices with Text Promptcited by 48
  27. 2024Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesiscited by 48
  28. 2025MegaTTS 3: Sparse Alignment Enhanced Latent Diffusion Transformer for Zero-Shot Speech Synthesiscited by 41
  29. 2025Scaling Video-Language Models to 10K Frames via Hierarchical Differential Distillationcited by 37
  30. 2024Autoregressive Diffusion Transformer for Text-to-Speech Synthesiscited by 34
  31. 2022StyleTTS: A Style-Based Generative Model for Natural and Diverse Text-to-Speech Synthesiscited by 33
  32. 2024DiTTo-TTS: Diffusion Transformers for Scalable Text-to-Speech without Domain-Specific Factorscited by 32
  33. 2024Emo-DPO: Controllable Emotional Speech Synthesis through Direct Preference Optimizationcited by 32
  34. 2025OmniSync: Towards Universal Lip Synchronization via Diffusion Transformerscited by 27
  35. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 25
  36. 2020Transformer Transducer: One Model Unifying Streaming and Non-streaming Speech Recognitioncited by 24
  37. 2023Fine-grained Audio-Visual Joint Representations for Multimodal Large Language Modelscited by 21
  38. 2025Kling-Avatar: Grounding Multimodal Instructions for Cascaded Long-Duration Avatar Animation Synthesiscited by 21

50 most cited of 64 All 64 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks
  2. 2025InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling
  3. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks
  4. 2024A Survey of Multimodal Large Language Model from A Data-centric Perspective
  5. 2024The Revolution of Multimodal Large Language Models: A Survey
  6. 2024Exploring the Design Space of Visual Context Representation in Video MLLMs
  7. 2024Model Composition for Multimodal Large Language Models
  8. 2025Some Modalities are More Equal Than Others: Decoding and Architecting Multimodal Integration in MLLMs