mlsanityData to 7 Oct 2026

Aria: An Open Multimodal Native Mixture-of-Experts Model

arXiv 2410.05993 ↗First posted 8 Oct 2024cs.CVcited by 136

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Information comes in diverse modalities. Multimodal native AI models are essential to integrate real-world information and deliver comprehensive understanding. While proprietary multimodal native models exist, their lack of openness imposes obstacles for adoptions, let alone adaptations. To fill this gap, we introduce Aria, an open multimodal native model with best-in-class performance across a wide range of multimodal, language, and coding tasks. Aria is a mixture-of-expert model with 3.9B and 3.5B activated parameters per visual token and text token, respectively. It outperforms Pixtral-12B and Llama3.2-11B, and is competitive against the best proprietary models on various multimodal tasks. We pre-train Aria from scratch following a 4-stage pipeline, which progressively equips the model with strong capabilities in language understanding, multimodal understanding, long context window, and instruction following. We open-source the model weights along with a codebase that facilitates easy adoptions and adaptations of Aria in real-world applications.

Key facts. Cited by 136 papers in the map (data through 7 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 23 of its 24 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

23 of 24 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts31 Jul
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan

Other references · 21

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,544
  12. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
9 more
  1. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  2. 2024LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understandingcited by 644
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  5. 2024What matters when building vision-language models?cited by 321
  6. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 226
  7. 2024Scaling Laws for Fine-Grained Mixture of Expertscited by 166
  8. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  9. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 97; 2026: 22.

136 papers.

Sort:

By area:

  1. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  2. 2025VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understandingcited by 502
  3. 2025Kimi-VL Technical Reportcited by 381
  4. 2024Large Language Model-Brained GUI Agents: A Surveycited by 160
  5. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 154
  6. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  7. 2024Video-RAG: Visually-aligned Retrieval-Augmented Long Video Comprehensioncited by 134
  8. 2025MMVU: Measuring Expert-Level Multi-Discipline Video Understandingcited by 131
  9. 2025MMSearch-R1: Incentivizing LMMs to Searchcited by 120
  10. 2025Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Searchcited by 116
  11. 2025Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understandingcited by 80
  12. 2025MotionBench: Benchmarking and Improving Fine-grained Video Motion Understanding for Vision Language Modelscited by 69
38 more
  1. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  2. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  3. 2025Detect Anything via Next Point Predictioncited by 66
  4. 2025Eagle 2.5: Boosting Long-Context Post-Training for Frontier Vision-Language Modelscited by 62
  5. 2025LVAgent: Long Video Understanding by Multi-Round Dynamical Collaboration of MLLM Agentscited by 58
  6. 2024InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactionscited by 48
  7. 2025Scaling Laws for Native Multimodal Modelscited by 47
  8. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 43
  9. 2025VCR-Bench: A Comprehensive Evaluation Framework for Video Chain-of-Thought Reasoningcited by 41
  10. 2025Multimodal RewardBench: Holistic Evaluation of Reward Models for Vision Language Modelscited by 40
  11. 2024WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisinescited by 39
  12. 2025Long-VITA: Scaling Large Multi-modal Models to 1 Million Tokens with Leading Short-Context Accuracycited by 39
  13. 2025EVEv2: Improved Baselines for Encoder-Free Vision-Language Modelscited by 38
  14. 2025Video-MMLU: A Massive Multi-Discipline Lecture Understanding Benchmarkcited by 37
  15. 2024Lost in Time: A New Temporal Benchmark for VideoLLMscited by 36
  16. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  17. 2025Generative Frame Sampler for Long Video Understandingcited by 35
  18. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  19. 2024ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understandingcited by 33
  20. 2026Beyond Language Modeling: An Exploration of Multimodal Pretrainingcited by 32
  21. 2025PHYSICS: Benchmarking Foundation Models on University-Level Physics Problem Solvingcited by 31
  22. 2026LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligencecited by 31
  23. 2024ChatRex: Taming Multimodal LLM for Joint Perception and Understandingcited by 30
  24. 2025FOCUS: Efficient Keyframe Selection for Long Video Understandingcited by 30
  25. 2025Temporal Preference Optimization for Long-Form Video Understandingcited by 28
  26. 2025Seeing the Arrow of Time in Large Multimodal Modelscited by 28
  27. 2025FAVOR-Bench: A Comprehensive Benchmark for Fine-Grained Video Motion Understandingcited by 24
  28. 2025VRBench: A Benchmark for Multi-Step Reasoning in Long Narrative Videoscited by 24
  29. 2025Rex-Thinker: Grounded Object Referring via Chain-of-Thought Reasoningcited by 23
  30. 2025VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videoscited by 23
  31. 2025VidCapBench: A Comprehensive Benchmark of Video Captioning for Controllable Text-to-Video Generationcited by 22
  32. 2025VideoLucy: Deep Memory Backtracking for Long Video Understandingcited by 21
  33. 2025Referring to Any Personcited by 20
  34. 2025HiMTok: Learning Hierarchical Mask Tokens for Image Segmentation with Large Multimodal Modelcited by 19
  35. 2024VideoAutoArena: An Automated Arena for Evaluating Large Multimodal Models in Video Analysis through User Simulationcited by 18
  36. 2025Uni-X: Mitigating Modality Conflict with a Two-End-Separated Architecture for Unified Multimodal Modelscited by 18
  37. 2025FlexSelect: Flexible Token Selection for Efficient Long Video Understandingcited by 17
  38. 2025IR3D-Bench: Evaluating Vision-Language Model Scene Understanding as Agentic Inverse Renderingcited by 17

50 most cited of 136 All 136 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models
  2. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
  3. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  4. 2026Toward Native Multimodal Modeling: A Roadmap
  5. 2026MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
  6. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts
  7. 2025Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
  8. 2025OpenUni: A Simple Baseline for Unified Multimodal Understanding and Generation