# Deep Bayesian Future Fusion for Self-Supervised, High-Resolution, Off-Road Mapping

arXiv: https://arxiv.org/abs/2403.11876 · HTML: https://mlsanity.com/paper/2403.11876/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 48 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 30 of 34)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers](https://mlsanity.com/paper/2105.15203/) (2021)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Training Generative Adversarial Networks with Limited Data](https://mlsanity.com/paper/2006.06676/) (2020)
- [Free-Form Image Inpainting with Gated Convolution](https://mlsanity.com/paper/1806.03589/) (2018)
- [Image Inpainting for Irregular Holes Using Partial Convolutions](https://mlsanity.com/paper/1804.07723/) (2018)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [A U-Net Based Discriminator for Generative Adversarial Networks](https://mlsanity.com/paper/2002.12655/) (2020)
- [Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting](https://mlsanity.com/paper/2005.09704/) (2020)
- [TartanVO: A Generalizable Learning-based VO](https://mlsanity.com/paper/2011.00359/) (2020)
- [CompletionFormer: Depth Completion with Convolutions and Vision Transformers](https://mlsanity.com/paper/2304.13030/) (2023)
- [How do neural networks see depth in single images?](https://mlsanity.com/paper/1905.07005/) (2019)
- [RigNet: Repetitive Image Guided Network for Depth Completion](https://mlsanity.com/paper/2107.13802/) (2021)
- [Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting](https://mlsanity.com/paper/2302.13130/) (2023)
- [Dynamic Spatial Propagation Network for Depth Completion](https://mlsanity.com/paper/2202.09769/) (2022)
- [Neural Texture Extraction and Distribution for Controllable Person Image Synthesis](https://mlsanity.com/paper/2204.06160/) (2022)
- [GANav: Efficient Terrain Segmentation for Robot Navigation in Unstructured Outdoor Environments](https://mlsanity.com/paper/2103.04233/) (2021)
- [TartanDrive: A Large-Scale Dataset for Learning Off-Road Dynamics Models](https://mlsanity.com/paper/2205.01791/) (2022)
- [Bidirectional Attention Network for Monocular Depth Estimation](https://mlsanity.com/paper/2009.00743/) (2020)
- [Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks](https://mlsanity.com/paper/2110.04076/) (2021)
- [TerrainNet: Visual Modeling of Complex Terrain for High-speed, Off-road Navigation](https://mlsanity.com/paper/2303.15771/) (2023)
- [How Does It Feel? Self-Supervised Costmap Learning for Off-Road Vehicle Traversability](https://mlsanity.com/paper/2209.10788/) (2022)
- [Super Odometry: IMU-centric LiDAR-Visual-Inertial Estimator for Challenging Environments](https://mlsanity.com/paper/2104.14938/) (2021)
- [RoadRunner -- Learning Traversability Estimation for Autonomous Off-road Driving](https://mlsanity.com/paper/2402.19341/) (2024)

## Built on by (most cited first, 3 of 3)

- [MapEx: Indoor Structure Exploration with Probabilistic Information Gain from Global Map Predictions](https://mlsanity.com/paper/2409.15590/) (2024)
- [Map It Anywhere (MIA): Empowering Bird's Eye View Mapping using Large-scale Public Data](https://mlsanity.com/paper/2407.08726/) (2024)
- [RoadRunner M&M -- Learning Multi-range Multi-resolution Traversability Maps for Autonomous Off-road Navigation](https://mlsanity.com/paper/2409.10940/) (2024)

## Abstract (arXiv)

High-speed off-road navigation requires long-range, high-resolution maps to enable robots to safely navigate over different surfaces while avoiding dangerous obstacles. However, due to limited computational power and sensing noise, most approaches to off-road mapping focus on producing coarse (20-40cm) maps of the environment. In this paper, we propose Future Fusion, a framework capable of generating dense, high-resolution maps from sparse sensing data (30m forward at 2cm). This is accomplished by - (1) the efficient realization of the well-known Bayes filtering within the standard deep learning models that explicitly accounts for the sparsity pattern in stereo and LiDAR depth data, and (2) leveraging perceptual losses common in generative image completion. The proposed methodology outperforms the conventional baselines. Moreover, the learned features and the completed dense maps lead to improvements in the downstream navigation task.