# The Foreseeable Future: Self-Supervised Learning to Predict Dynamic Scenes for Indoor Navigation

arXiv: https://arxiv.org/abs/2208.12602 · HTML: https://mlsanity.com/paper/2208.12602/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Aug 2022.

## Built on by (most cited first, 5 of 5)

- [DR-MPC: Deep Residual Model Predictive Control for Real-world Social Navigation](https://mlsanity.com/paper/2410.10646/) (2024)
- [Perpetua: Multi-Hypothesis Persistence Modeling for Semi-Static Environments](https://mlsanity.com/paper/2507.18808/) (2025)
- [TRANS: Terrain-aware Reinforcement Learning for Agile Navigation of Quadruped Robots under Social Interactions](https://mlsanity.com/paper/2602.12724/) (2026)
- [ST-Booster: An Iterative SpatioTemporal Perception Booster for Vision-and-Language Navigation in Continuous Environments](https://mlsanity.com/paper/2504.09843/) (2025)
- [Deep Learning Perspective of Scene Understanding in Autonomous Robots](https://mlsanity.com/paper/2512.14020/) (2025)

## Abstract (arXiv)

We present a method for generating, predicting, and using Spatiotemporal Occupancy Grid Maps (SOGM), which embed future semantic information of real dynamic scenes. We present an auto-labeling process that creates SOGMs from noisy real navigation data. We use a 3D-2D feedforward architecture, trained to predict the future time steps of SOGMs, given 3D lidar frames as input. Our pipeline is entirely self-supervised, thus enabling lifelong learning for real robots. The network is composed of a 3D back-end that extracts rich features and enables the semantic segmentation of the lidar frames, and a 2D front-end that predicts the future information embedded in the SOGM representation, potentially capturing the complexities and uncertainties of real-world multi-agent, multi-future interactions. We also design a navigation system that uses these predicted SOGMs within planning, after they have been transformed into Spatiotemporal Risk Maps (SRMs). We verify our navigation system's abilities in simulation, validate it on a real robot, study SOGM predictions on real data in various circumstances, and provide a novel indoor 3D lidar dataset, collected during our experiments, which includes our automated annotations.