mlsanityData to 14 Oct 2026

VeOmni: Scaling Any Modality Model Training with Model-Centric Distributed Recipe Zoo

arXiv 2508.02317 ↗First posted 4 Aug 2025cs.CL cs.AI cs.DCcited by 20

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Recent advances in large language models (LLMs) have driven impressive progress in omni-modal understanding and generation. However, training omni-modal LLMs remains a significant challenge due to the heterogeneous model architectures required to process diverse modalities, necessitating sophisticated system design for efficient large-scale training. Existing frameworks typically entangle model definition with parallel logic, incurring limited scalability and substantial engineering overhead for end-to-end omni-modal training. We present VeOmni, a modular and efficient training framework to accelerate the development of omni-modal LLMs. VeOmni introduces model-centric distributed recipes that decouples communication from computation, enabling efficient 3D parallelism on omni-modal LLMs. VeOmni also features a flexible configuration interface supporting seamless integration of new modalities with minimal code change. Using VeOmni, a omni-modal mixture-of-experts (MoE) model with 30B parameters can be trained with over 2,800 tokens/sec/GPU throughput and scale to 160K context lengths via 3D parallelism on 128 GPUs, showcasing its superior efficiency and scalability for training large omni-modal LLMs.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 66 of its 78 references that are in the map. First posted on arXiv 4 Aug 2025.

Builds on

66 of 78 references.

References · 66

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2023Visual Instruction Tuningcited by 7,869
  8. 2022Generative Adversarial Networkscited by 5,627
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  12. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
38 more
  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  7. 2023Video-LLaVA: Learning United Visual Representation by Alignment Before Projectioncited by 1,402
  8. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  9. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,175
  11. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  12. 2024Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Predictioncited by 1,023
  13. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,005
  14. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  15. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  16. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  17. 2024Autoregressive Model Beats Diffusion: Llama for Scalable Image Generationcited by 777
  18. 2024Show-o: One Single Transformer to Unify Multimodal Understanding and Generationcited by 752
  19. 2025Qwen2.5-Omni Technical Reportcited by 736
  20. 2025Vision-R1: Incentivizing Reasoning Capability in Multimodal Large Language Modelscited by 665
  21. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  22. 2024Autoregressive Image Generation without Vector Quantizationcited by 651
  23. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  24. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 519
  25. 2024Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Modelcited by 483
  26. 2024Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generationcited by 457
  27. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  28. 2025Seedance 1.0: Exploring the Boundaries of Video Generation Modelscited by 259
  29. 2025MMaDA: Multimodal Large Diffusion Language Modelscited by 250
  30. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  31. 2024Mini-Omni: Language Models Can Hear, Talk While Thinking in Streamingcited by 210
  32. 2025Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontierscited by 199
  33. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 198
  34. 2023SmartEdit: Exploring Complex Instruction-based Image Editing with Multimodal Large Language Modelscited by 177
  35. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  36. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  37. 2024Liger Kernel: Efficient Triton Kernels for LLM Trainingcited by 130
  38. 2022MoVQ: Modulating Quantized Vectors for High-Fidelity Image Generationcited by 125

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 16.

20 papers.

Sort:

By area:

  1. 2026A Pragmatic VLA Foundation Modelcited by 103
  2. 2026Timer-S1: A Billion-Scale Time Series Foundation Model with Serial Scalingcited by 28
  3. 2025LongCat-Flash-Omni Technical Reportcited by 26
  4. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 24
  5. 2026SWE-MeM: Learning Adaptive Memory Management for Long-Horizon Coding Agentscited by 15
  6. 2026Scaling Mixture-of-Experts Video Pretraining for Embodied Intelligencecited by 15
  7. 2026LLaDA2.0-Uni: Unifying Multimodal Understanding and Generation with Diffusion Large Language Modelcited by 13
  8. 2025OpenHA: A Series of Open-Source Hierarchical Agentic Models in Minecraftcited by 10
  9. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attentioncited by 5
  10. 2026Visual Generation in the New Era: An Evolution from Atomic Mapping to Agentic World Modelingcited by 5
  11. 2026Multi-Block Diffusion Language Modelscited by 3
  12. 2025SAIL-RL: Guiding MLLMs in When and How to Think via Dual-Reward RL Tuningcited by 2
8 more
  1. 2026SAS: Simple Attention Sparsification via End-to-End Optimization of Context Rankingcited by 2
  2. 2026veScale-FSDP: Flexible and High-Performance FSDP at Scalecited by 1
  3. 2026TransVLM: A Vision-Language Framework and Benchmark for Detecting Any Shot Transitionscited by 1
  4. 2026Training Long-Context Vision-Language Models Effectively with Generalization Beyond 128K Contextcited by 1
  5. 2026Charon: A Unified and Fine-Grained Simulator for Large-Scale LLM Training and Inferencecited by 0
  6. 2026LLaDA-UI: Bringing Block-wise Diffusion to Vision-Language GUI Agentscited by 0
  7. 2026OmniFysics-Nano-V2 Technical Report: Understanding the Physical World Across Modalitiescited by 0
  8. 2026Pistis Technical Reportcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025M2-omni: Advancing Omni-MLLM for Comprehensive Modality Support with Competitive Performance
  2. 2025Uni-MoE-2.0-Omni: Scaling Language-Centric Omnimodal Large Model with Advanced MoE, Training and Data
  3. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
  4. 2024Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
  5. 2025Training-Free Multimodal Large Language Model Orchestration
  6. 2026Heterogeneous Parallelism for Multimodal Large Language Model Training
  7. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  8. 2024OmniBench: Towards The Future of Universal Omni-Language Models