mlsanityData to 7 Oct 2026

SAME: Learning Generic Language-Guided Visual Navigation with State-Adaptive Mixture of Experts

arXiv 2412.05552 ↗First posted 7 Dec 2024cs.CV cs.AI cs.CLcited by 16

On the map: Mixture-of-experts → MoE in reinforcement learning and robotics

Abstract (arXiv)

The academic field of learning instruction-guided visual navigation can be generally categorized into high-level category-specific search and low-level language-guided navigation, depending on the granularity of language instruction, in which the former emphasizes the exploration process, while the latter concentrates on following detailed textual commands. Despite the differing focuses of these tasks, the underlying requirements of interpreting instructions, comprehending the surroundings, and inferring action decisions remain consistent. This paper consolidates diverse navigation tasks into a unified and generic framework -- we investigate the core difficulties of sharing general knowledge and exploiting task-specific capabilities in learning navigation and propose a novel State-Adaptive Mixture of Experts (SAME) model that effectively enables an agent to infer decisions based on different-granularity language and dynamic observations. Powered by SAME, we present a versatile agent capable of addressing seven navigation tasks simultaneously that outperforms or achieves highly comparable performance to task-specific agents.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Placed in MoE in reinforcement learning and robotics, one of 12 directions in Mixture-of-experts. Builds on 106 of its 126 references that are in the map. First posted on arXiv 7 Dec 2024.

Builds on

106 of 126 references.

References · 106

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2024Mixtral of Expertscited by 1,785
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  6. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  7. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  10. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  11. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
38 more
  1. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  4. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  5. 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 579
  6. 2018On Evaluation of Embodied Navigation Agentscited by 526
  7. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  8. 2020Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environmentscited by 429
  9. 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 421
  10. 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 401
  11. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  12. 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 369
  13. 2019DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Framescited by 352
  14. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  15. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 337
  16. 2019REVERIE: Remote Embodied Visual Referring Expression in Real Indoor Environmentscited by 330
  17. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 304
  18. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 300
  19. 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 274
  20. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  21. 2019Vision-and-Dialog Navigationcited by 246
  22. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 241
  23. 2020ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objectscited by 240
  24. 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 228
  25. 2024NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigationcited by 222
  26. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  27. 2022CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigationcited by 194
  28. 2020Neural Topological SLAM for Visual Navigationcited by 193
  29. 2020RoboTHOR: An Open Simulation-to-Real Embodied AI Platformcited by 190
  30. 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 187
  31. 2024OpenMoE: An Early Effort on Open Mixture-of-Experts Language Modelscited by 185
  32. 2019Vision-Language Navigation with Self-Supervised Auxiliary Reasoning Taskscited by 177
  33. 2020Improving Vision-and-Language Navigation with Image-Text Pairs from the Webcited by 177
  34. 2019Self-Monitoring Navigation Agent via Auxiliary Progress Estimationcited by 173
  35. 2023ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environmentscited by 165
  36. 2023Habitat Synthetic Scenes Dataset (HSSD-200): An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigationcited by 162
  37. 2024InstructNav: Zero-shot System for Generic Instruction Navigation in Unexplored Environmentcited by 150
  38. 2022PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learningcited by 149

50 most cited of 106 All 106 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 10.

16 papers.

Sort:

By area:

  1. 2024Bootstrapping Language-Guided Navigation Learning with Self-Refining Data Flywheelcited by 18
  2. 2025VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluationcited by 17
  3. 2026Qwen-RobotNav Technical Report: A Scalable Navigation Model Designed for an Agentic Navigation Systemcited by 10
  4. 2025CoNav: Collaborative Cross-Modal Reasoning for Embodied Navigationcited by 5
  5. 2025VLN-MME: Diagnosing MLLMs as Language-guided Visual Navigation agentscited by 5
  6. 2025Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Expertscited by 4
  7. 2025Breaking Down and Building Up: Mixture of Skill-Based Vision-and-Language Navigation Agentscited by 4
  8. 2026VLM-Loc: Localization in Point Cloud Maps via Vision-Language Modelscited by 4
  9. 2026Lifelong Embodied Navigation Learningcited by 3
  10. 2026Ask When It Pays: Cost-Aware Open-Ended Interaction for Instance Goal Navigationcited by 2
  11. 2026A Comprehensive Survey and Systematic Real-World Evaluation of Embodied Vision-and-Language Navigationcited by 1
  12. 2026Agentic Game Development as a Verifiable Trajectory Data Engine for Scaling World Modelscited by 1
4 more
  1. 2026Scaffolding Foundation Models into Physical-World Agents Pushes the Frontier of Long-Horizon Navigationcited by 1
  2. 2026Implicit Geometry Representations for Vision-and-Language Navigation from Web Videoscited by 0
  3. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Surveycited by 0
  4. 2026Beyond Waypoints: A Trajectory-Centric Waypointing Paradigm for Vision-Language Navigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Learning to Retrieve Navigable Candidates for Efficient Vision-and-Language Navigation
  2. 2024NavGPT-2: Unleashing Navigational Reasoning Capability for Large Vision-Language Models
  3. 2020Evolving Graphical Planner: Contextual Global Planning for Vision-and-Language Navigation
  4. 2025CompassNav: Steering From Path Imitation To Decision Understanding In Navigation
  5. 2025Run, Ruminate, and Regulate: A Dual-process Thinking System for Vision-and-Language Navigation
  6. 2025Landmark-Guided Knowledge for Vision-and-Language Navigation
  7. 2026What do VLM-Based Vision-Language Navigation Models Rely on: Interpreting and Steering Policy Behavior
  8. 2025Fine-Grained Instruction-Guided Graph Reasoning for Vision-and-Language Navigation