mlsanityData to 7 Oct 2026

MoE3D: Mixture of Experts meets Multi-Modal 3D Understanding

arXiv 2511.22103 ↗First posted 27 Nov 2025cs.CVcited by 6

On the map: Mixture-of-experts → Vision mixture-of-experts

Abstract (arXiv)

Multi-modal 3D understanding is a fundamental task in computer vision. Previous multi-modal fusion methods typically employ a single, dense fusion network, struggling to handle the significant heterogeneity and complexity across modalities, leading to suboptimal performance. In this paper, we propose MoE3D, which integrates Mixture of Experts (MoE) into the multi-modal learning framework. The core is that we deploy a set of specialized "expert" networks, each adept at processing a specific modality or a mode of cross-modal interaction. Specifically, the MoE-based transformer is designed to better utilize the complementary information hidden in the visual features. Information aggregation module is put forward to further enhance the fusion performance. Top-1 gating is employed to make one expert process features with expert groups, ensuring high efficiency. We further propose a progressive pre-training strategy to better leverage the semantic and 2D prior, thus equipping the network with good initialization. Our MoE3D achieves competitive performance across four prevalent 3D understanding tasks. Notably, our MoE3D surpasses the top-performing counterpart by 6.1 mIoU on Multi3DRefer.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Vision mixture-of-experts, one of 12 directions in Mixture-of-experts. Builds on 55 of its 73 references that are in the map. First posted on arXiv 27 Nov 2025.

Builds on

55 of 73 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022A Review of Sparse Expert Models in Deep Learning4 Sep
  2. 2021Scaling Vision with Sparse Mixture of Experts10 Jun

Other references · 53

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2024Mixtral of Expertscited by 1,785
  5. 20194D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networkscited by 1,201
  6. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  7. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,093
  8. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  9. 2023LISA: Reasoning Segmentation via Large Language Modelcited by 770
  10. 2022Language-driven Semantic Segmentationcited by 711
  11. 2017Frustum PointNets for 3D Object Detection from RGB-D Datacited by 640
  12. 2022OpenScene: 3D Scene Understanding with Open Vocabulariescited by 556
38 more
  1. 2019PointPainting: Sequential Fusion for 3D Object Detectioncited by 449
  2. 2019ScanRefer: 3D Object Localization in RGB-D Scans using Natural Languagecited by 446
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  4. 2017Large-scale Point Cloud Semantic Segmentation with Superpoint Graphscited by 384
  5. 2023An Embodied Generalist Agent in 3D Worldcited by 355
  6. 2021ScanQA: 3D Question Answering for Spatial Scene Understandingcited by 340
  7. 2023PointLLM: Empowering Large Language Models to Understand Point Cloudscited by 326
  8. 2022SQA3D: Situated Question Answering in 3D Scenescited by 303
  9. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
  10. 2022Language-Grounded Indoor 3D Semantic Segmentation in the Wildcited by 243
  11. 2020Scan2Cap: Context-aware Dense Captioning in RGB-D Scanscited by 215
  12. 2023LL3DA: Visual Interactive Instruction Tuning for Omni-3D Understanding, Reasoning, and Planningcited by 206
  13. 20233D-VisTA: Pre-trained Transformer for 3D Vision and Text Alignmentcited by 201
  14. 2022Tutel: Adaptive Mixture-of-Experts at Scalecited by 199
  15. 2023Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Followingcited by 192
  16. 20203D-CVF: Generating Joint Camera and LiDAR Features Using Cross-View Spatial Feature Fusion for 3D Object Detectioncited by 176
  17. 2023Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifierscited by 170
  18. 2024Scene-LLM: Extending Language Model for 3D Visual Understanding and Reasoningcited by 170
  19. 2019MVX-Net: Multimodal VoxelNet for 3D Object Detectioncited by 162
  20. 2020CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detectioncited by 147
  21. 2023EmbodiedScan: A Holistic Multi-Modal 3D Perception Suite Towards Embodied AIcited by 146
  22. 2023Multi3DRefer: Grounding Text Description to Multiple 3D Objectscited by 142
  23. 2024Grounded 3D-LLM with Referent Tokenscited by 111
  24. 2023OneFormer3D: One Transformer for Unified Point Cloud Segmentationcited by 95
  25. 2023LoGoNet: Towards Accurate 3D Object Detection with Local-to-Global Cross-Modal Fusioncited by 80
  26. 2023End-to-End 3D Dense Captioning with Vote2Cap-DETRcited by 77
  27. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  28. 20253D-LLaVA: Towards Generalist 3D LMMs with Omni Superpoint Transformercited by 67
  29. 2021Multi-Modal 3D Object Detection in Autonomous Driving: a Surveycited by 63
  30. 2023UniSeg: A Unified Multi-Modal LiDAR Segmentation Network and the OpenPCSeg Codebasecited by 57
  31. 2021D3Net: A Unified Speaker-Listener Architecture for 3D Dense Captioning and Visual Groundingcited by 52
  32. 2022MORE: Multi-Order RElation Mining for Dense Captioning in 3D Scenescited by 46
  33. 2023Mask-Attention-Free Transformer for 3D Instance Segmentationcited by 45
  34. 2022Spatiality-guided Transformer for 3D Dense Captioning on Point Cloudscited by 40
  35. 20233D-STMN: Dependency-Driven Superpoint-Text Matching Network for End-to-End 3D Referring Expression Segmentationcited by 36
  36. 2023Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networkscited by 35
  37. 2024SegPoint: Segment Any Point Cloud via Large Language Modelcited by 34
  38. 2024M4oE: A Foundation Model for Medical Multimodal Image Segmentation with Mixture of Expertscited by 26

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Segment and Select: Vision-Language Segmentation in 3D Scenarioscited by 2
  2. 2026Weather-Conditioned Branch Routing for Robust LiDAR-Radar 3D Object Detectioncited by 1
  3. 2026LER-YOLO: Reliability-Aware Expert Routing for Misaligned RGB-Infrared UAV Detectioncited by 0
  4. 2026Heterogeneous and Adept Snapshot Distillation for 3D Semantic Segmentationcited by 0
  5. 2026CMTFormer: Marrying Transformer with Hierarchical Information Interaction for RGB-Event Object Detectioncited by 0
  6. 2026PVCap: Towards Accurate 3D Dense Captioning via PseudoCap and VoxelCapNetcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Uni3D-MoE: Scalable Multimodal 3D Scene Understanding via Mixture of Experts
  2. 2024Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization
  3. 2025MoRE: 3D Visual Geometry Reconstruction Meets Mixture-of-Experts
  4. 2024ViMoE: An Empirical Study of Designing Vision Mixture-of-Experts
  5. 2025Point-MoE: Large-Scale Multi-Dataset Training with Mixture-of-Experts for 3D Semantic Segmentation
  6. 2024Mixture of Experts in Image Classification: What's the Sweet Spot?
  7. 2026MoE3D: A Mixture-of-Experts Module for 3D Reconstruction
  8. 20253D-MoE: Towards Spatial Intelligence with Mixture-of-Experts for 3D Reasoning and Action Generation