# Situational Fusion of Visual Representation for Visual Navigation

arXiv: https://arxiv.org/abs/1908.09073 · HTML: https://mlsanity.com/paper/1908.09073/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Aug 2019.

## Built on by (most cited first, 28 of 28)

- [Bird's-Eye-View Scene Graph for Vision-Language Navigation](https://mlsanity.com/paper/2308.04758/) (2023)
- [Zero Experience Required: Plug & Play Modular Transfer Learning for Semantic Visual Navigation](https://mlsanity.com/paper/2202.02440/) (2022)
- [Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring](https://mlsanity.com/paper/2106.03427/) (2021)
- [VTNet: Visual Transformer Network for Object Goal Navigation](https://mlsanity.com/paper/2105.09447/) (2021)
- [Learning to See before Learning to Act: Visual Pre-training for Manipulation](https://mlsanity.com/paper/2107.00646/) (2021)
- [Visual Navigation with Spatial Attention](https://mlsanity.com/paper/2104.09807/) (2021)
- [Adversarial Bayesian Augmentation for Single-Source Domain Generalization](https://mlsanity.com/paper/2307.09520/) (2023)
- [Learning Active Camera for Multi-Object Navigation](https://mlsanity.com/paper/2210.07505/) (2022)
- [Visual Representation Learning for Preference-Aware Path Planning](https://mlsanity.com/paper/2109.08968/) (2021)
- [Improving Diversity with Adversarially Learned Transformations for Domain Generalization](https://mlsanity.com/paper/2206.07736/) (2022)
- [Environment Predictive Coding for Embodied Agents](https://mlsanity.com/paper/2102.02337/) (2021)
- [From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning](https://mlsanity.com/paper/2507.22028/) (2025)
- [MaAST: Map Attention with Semantic Transformersfor Efficient Visual Navigation](https://mlsanity.com/paper/2103.11374/) (2021)
- [Agent-Centric Relation Graph for Object Visual Navigation](https://mlsanity.com/paper/2111.14422/) (2021)
- [Cross-Episodic Curriculum for Transformer Agents](https://mlsanity.com/paper/2310.08549/) (2023)
- [Aligning Knowledge Graph with Visual Perception for Object-goal Navigation](https://mlsanity.com/paper/2402.18892/) (2024)
- [Goal-Aware Cross-Entropy for Multi-Target Reinforcement Learning](https://mlsanity.com/paper/2110.12985/) (2021)
- [GraphMapper: Efficient Visual Navigation by Scene Graph Generation](https://mlsanity.com/paper/2205.08325/) (2022)
- [VANP: Learning Where to See for Navigation with Self-Supervised Vision-Action Pre-Training](https://mlsanity.com/paper/2403.08109/) (2024)
- [Multisource Collaborative Domain Generalization for Cross-Scene Remote Sensing Image Classification](https://mlsanity.com/paper/2412.03897/) (2024)
- [Deep Learning for Embodied Vision Navigation: A Survey](https://mlsanity.com/paper/2108.04097/) (2021)
- [Image-based Navigation in Real-World Environments via Multiple Mid-level Representations: Fusion Models, Benchmark and Efficient Evaluation](https://mlsanity.com/paper/2202.01069/) (2022)
- [Comparison of Model-Free and Model-Based Learning-Informed Planning for PointGoal Navigation](https://mlsanity.com/paper/2212.08801/) (2022)
- [Leveraging Large Language Model-based Room-Object Relationships Knowledge for Enhancing Multimodal-Input Object Goal Navigation](https://mlsanity.com/paper/2403.14163/) (2024)
- [Symmetry-aware Neural Architecture for Embodied Visual Navigation](https://mlsanity.com/paper/2112.09515/) (2021)
- [Predicting Dense and Context-aware Cost Maps for Semantic Robot Navigation](https://mlsanity.com/paper/2210.08952/) (2022)
- [Pseudo-Trilateral Adversarial Training for Domain Adaptive Traversability Prediction](https://mlsanity.com/paper/2306.14370/) (2023)
- [UAS Visual Navigation in Large and Unseen Environments via a Meta Agent](https://mlsanity.com/paper/2503.15781/) (2025)

## Abstract (arXiv)

A complex visual navigation task puts an agent in different situations which call for a diverse range of visual perception abilities. For example, to "go to the nearest chair", the agent might need to identify a chair in a living room using semantics, follow along a hallway using vanishing point cues, and avoid obstacles using depth. Therefore, utilizing the appropriate visual perception abilities based on a situational understanding of the visual environment can empower these navigation models in unseen visual environments. We propose to train an agent to fuse a large set of visual representations that correspond to diverse visual perception abilities. To fully utilize each representation, we develop an action-level representation fusion scheme, which predicts an action candidate from each representation and adaptively consolidate these action candidates into the final action. Furthermore, we employ a data-driven inter-task affinity regularization to reduce redundancies and improve generalization. Our approach leads to a significantly improved performance in novel environments over ImageNet-pretrained baseline and other fusion methods.