mlsanityData to 14 Oct 2026

Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts

arXiv 2505.21079 ↗First posted 27 May 2025cs.CVcited by 4

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Recent advancements in multimodal large language models (MLLMs) have demonstrated considerable potential for comprehensive 3D scene understanding. However, existing approaches typically utilize only one or a limited subset of 3D modalities, resulting in incomplete representations of 3D scenes and reduced interpretive accuracy. Furthermore, different types of queries inherently depend on distinct modalities, indicating that uniform processing of all modality tokens may fail to effectively capture query-specific context. To address these challenges, we propose Uni3D-MoE, a sparse Mixture-of-Experts (MoE)-based 3D MLLM designed to enable adaptive 3D multimodal fusion. Specifically, Uni3D-MoE integrates a comprehensive set of 3D modalities, including multi-view RGB and depth images, bird's-eye-view (BEV) maps, point clouds, and voxel representations. At its core, our framework employs a learnable routing mechanism within the sparse MoE-based large language model, dynamically selecting appropriate experts at the token level. Each expert specializes in processing multimodal tokens based on learned modality preferences, thus facilitating flexible collaboration tailored to diverse task-specific requirements. Extensive evaluations on standard 3D scene understanding benchmarks and specialized datasets demonstrate the efficacy of Uni3D-MoE.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 78 of its 93 references that are in the map. First posted on arXiv 27 May 2025.

Builds on

78 of 93 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Ross3D: Reconstructive Visual Instruction Tuning with 3D-Awareness2 Apr
  2. 20254D-Bench: Benchmarking Multi-modal Large Language Models for 4D Object Understanding22 Mar
  3. 20253UR-LLM: An End-to-End Multimodal Large Language Model for 3D Scene Understanding14 Jan
  4. 2025GPT4Scene: Understand 3D Scenes from Videos with Vision-Language Models2 Jan
  5. 2024Video-3D LLM: Learning Position-Aware Video Representation for 3D Scene Understanding30 Nov
  6. 2024LLaVA-3D: A Simple yet Effective Pathway to Empowering LMMs with 3D-awareness26 Sep
  7. 2024Kestrel: 3D Multimodal LLM for Part-Aware Grounded Description29 May
  8. 2024When LLMs step into the 3D World: A Survey and Meta-Analysis of 3D Tasks via Multi-modal Large Language Models16 May
  9. 2024MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors2 May
  10. 2023LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding21 Dec
  11. 2023GPT4Point: A Unified Framework for Point-Language Understanding and Generation5 Dec
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planning30 Nov
1 more
  1. 2023Vision-Language Pre-training with Object Contrastive Learning for 3D Scene Understanding18 May

Other references · 65

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  3. 2023Visual Instruction Tuningcited by 7,869
  4. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  5. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  6. 2017ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenescited by 3,117
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  9. 2022Mixture-of-Experts with Expert Choice Routingcited by 623
  10. 2024LLaVA-Video: Video Instruction Tuning With Synthetic Datacited by 583
  11. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  12. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 452
38 more
  1. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  2. 2023ConceptGraphs: Open-Vocabulary 3D Scene Graphs for Perception and Planningcited by 365
  3. 2023An Embodied Generalist Agent in 3D Worldcited by 365
  4. 2023DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenescited by 360
  5. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 346
  6. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 333
  7. 2022SQA3D: Situated Question Answering in 3D Scenescited by 314
  8. 2022Mask3D: Mask Transformer for 3D Semantic Instance Segmentationcited by 303
  9. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 300
  10. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  11. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 218
  12. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 204
  13. 2024Hierarchical Open-Vocabulary 3D Scene Graphs for Language-Grounded Robot Navigationcited by 196
  14. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 194
  15. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 187
  16. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 175
  17. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 175
  18. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  19. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 146
  20. 2021InstanceRefer: Cooperative Holistic Understanding for Visual Grounding on Point Clouds through Instance Multi-level Contextual Referringcited by 141
  21. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  22. 2024SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Codecited by 134
  23. 2024SceneVerse: Scaling 3D Vision-Language Learning for Grounded Scene Understandingcited by 133
  24. 2023Chat-3D: Data-efficiently Tuning Large Language Model for Universal Dialogue of 3D Scenescited by 129
  25. 2021Bottom Up Top Down Detection Transformers for Language Grounding in Images and Point Cloudscited by 128
  26. 2024LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-trainingcited by 123
  27. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 122
  28. 2024Grounded 3D-LLM with Referent Tokenscited by 115
  29. 20223D-SPS: Single-Stage 3D Visual Grounding via Referred Point Progressive Selectioncited by 110
  30. 2024SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigationcited by 109
  31. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Expertscited by 108
  32. 2022EDA: Explicit Text-Decoupling and Dense Alignment for 3D Visual Groundingcited by 99
  33. 2024Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agentscited by 91
  34. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  35. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 78
  36. 2022X-Trans2Cap: Cross-Modal Knowledge Transfer using Transformer for 3D Dense Captioningcited by 70
  37. 2024Unifying 3D Vision-Language Understanding via Promptable Queriescited by 70
  38. 2024Multi-Modal Data-Efficient 3D Scene Understanding for Autonomous Drivingcited by 54

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Mixture of Experts in Large Language Modelscited by 15
  2. 2026Modeling Cross-vision Synergy for Unified Large Vision Modelcited by 1
  3. 2025DoReMi: Bridging 3D Domains via Topology-Aware Domain-Representation Mixture of Expertscited by 0
  4. 2026STAR: A Spatial-Topology Aware Routing Framework for Generalizable 3D Scene Understandingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding
  2. 2025Unleashing the Multi-View Fusion Potential: Noise Correction in VLM for Open-Vocabulary 3D Scene Understanding
  3. 2025Lemon: A Unified and Scalable 3D Multimodal Model for Universal Spatial Understanding
  4. 2025Fast3D: Accelerating 3D Multi-modal Large Language Models for Efficient 3D Scene Understanding
  5. 20253DRS: MLLMs Need 3D-Aware Representation Supervision for Scene Understanding
  6. 2026SpaR3D-MoE: Adaptive 3D Spatial Reasoning from Sparse Views Meets Geometry-Inductive Mixture-of-Experts
  7. 2026EVA01: Unified Native 3D Understanding and Generation via Mixture-of-Transformers
  8. 20243DMIT: 3D Multi-modal Instruction Tuning for Scene Understanding