# Seeing Through Uncertainty: Free-Energy-Inspired Real-Time Adaptation for Robust Visual Navigation

arXiv: https://arxiv.org/abs/2403.01977 · HTML: https://mlsanity.com/paper/2403.01977/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 47 references that are in the map.
- First posted on arXiv 4 Mar 2024.

## Builds on (most cited first, 30 of 31)

- [Squeeze-and-Excitation Networks](https://mlsanity.com/paper/1709.01507/) (2017)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Instance Normalization: The Missing Ingredient for Fast Stylization](https://mlsanity.com/paper/1607.08022/) (2016)
- [Habitat: A Platform for Embodied AI Research](https://mlsanity.com/paper/1904.01201/) (2019)
- [Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation](https://mlsanity.com/paper/2002.08546/) (2020)
- [Multi-Stage Progressive Image Restoration](https://mlsanity.com/paper/2102.02808/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Gibson Env: Real-World Perception for Embodied Agents](https://mlsanity.com/paper/1808.10654/) (2018)
- [Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05143/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [Revisiting Batch Normalization For Practical Domain Adaptation](https://mlsanity.com/paper/1603.04779/) (2016)
- [Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images](https://mlsanity.com/paper/2011.10650/) (2020)
- [Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning](https://mlsanity.com/paper/1605.08104/) (2016)
- [GNM: A General Navigation Model to Drive Any Robot](https://mlsanity.com/paper/2210.03370/) (2022)
- [The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization](https://mlsanity.com/paper/2112.00463/) (2021)
- [DiffuseVAE: Efficient, Controllable and High-Fidelity Generation from Low-Dimensional Latents](https://mlsanity.com/paper/2201.00308/) (2022)
- [Deep active inference agents using Monte-Carlo methods](https://mlsanity.com/paper/2006.04176/) (2020)
- [Memory-Augmented Reinforcement Learning for Image-Goal Navigation](https://mlsanity.com/paper/2101.05181/) (2021)
- [Active Inference in Robotics and Artificial Agents: Survey and Challenges](https://mlsanity.com/paper/2112.01871/) (2021)
- [Deep Active Inference](https://mlsanity.com/paper/1709.02341/) (2017)
- [Video Test-Time Adaptation for Action Recognition](https://mlsanity.com/paper/2211.15393/) (2022)
- [Multimodal VAE Active Inference Controller](https://mlsanity.com/paper/2103.04412/) (2021)
- [Contrastive Active Inference](https://mlsanity.com/paper/2110.10083/) (2021)
- [Deep Active Inference for Partially Observable MDPs](https://mlsanity.com/paper/2009.03622/) (2020)
- [Adaptive robot body learning and estimation through predictive coding](https://mlsanity.com/paper/1805.03104/) (2018)
- [Active Inference for Integrated State-Estimation, Control, and Learning](https://mlsanity.com/paper/2005.05894/) (2020)
- [Free Energy Principle for State and Input Estimation of a Quadcopter Flying in Wind](https://mlsanity.com/paper/2109.12052/) (2021)
- [Adaptation through prediction: multisensory active inference torque control](https://mlsanity.com/paper/2112.06752/) (2021)
- [Leading or Following? Dyadic Robot Imitative Interaction Using the Active Inference Framework](https://mlsanity.com/paper/2103.02137/) (2021)
- [Unbiased Active Inference for Classical Control](https://mlsanity.com/paper/2207.13409/) (2022)

## Abstract (arXiv)

Navigation in the natural world is a feat of adaptive inference, where biological organisms maintain goal-directed behaviour despite noisy and incomplete sensory streams. Central to this ability is the Free Energy Principle (FEP), which posits that perception is a generative process where the brain minimises Variational Free Energy (VFE) to maintain accurate internal models of the world. While Deep Neural Networks (DNNs) have served as powerful analogues for biological brains, they typically lack the real-time plasticity required to handle abrupt sensory shifts. We introduce FEP-Nav, a biologically inspired framework for real-time perceptual adaptation in robust visual navigation. Motivated by the decomposition of VFE into prediction error and Bayesian surprise, FEP-Nav combines a Top-down Decoder, which provides an internal expectation of uncorrupted sensory input, with Adaptive Normalisation, which adjusts shifted feature distributions toward prior statistics. We interpret reconstruction and normalisation as approximate mechanisms for reducing the corresponding VFE-related terms during inference without gradient-based updates. Experiments across simulated and real-world visual corruptions show that FEP-Nav restores performance lost under visual corruption, outperforming non-adaptive baselines and strong adaptive methods. These results suggest that variational principles can provide a useful design perspective for robust autonomous behaviour under degraded sensory conditions.