# From Cognition to Precognition: A Future-Aware Framework for Social Navigation

arXiv: https://arxiv.org/abs/2409.13244 · HTML: https://mlsanity.com/paper/2409.13244/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 60 references that are in the map.
- First posted on arXiv 20 Sep 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Matterport3D: Learning from RGB-D Data in Indoor Environments](https://mlsanity.com/paper/1709.06158/) (2017)
- [Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning](https://mlsanity.com/paper/2108.10470/) (2021)
- [Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI](https://mlsanity.com/paper/2109.08238/) (2021)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames](https://mlsanity.com/paper/1911.00357/) (2019)
- [TNT: Target-driveN Trajectory Prediction](https://mlsanity.com/paper/2008.08294/) (2020)
- [Human Motion Trajectory Prediction: A Survey](https://mlsanity.com/paper/1905.06113/) (2019)
- [Habitat 3.0: A Co-Habitat for Humans, Avatars and Robots](https://mlsanity.com/paper/2310.13724/) (2023)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Crowd-Robot Interaction: Crowd-aware Robot Navigation with Attention-based Deep Reinforcement Learning](https://mlsanity.com/paper/1809.08835/) (2018)
- [From Goals, Waypoints & Paths To Long Term Human Trajectory Forecasting](https://mlsanity.com/paper/2012.01526/) (2020)
- [Socially Aware Motion Planning with Deep Reinforcement Learning](https://mlsanity.com/paper/1703.08862/) (2017)
- [Human Trajectory Forecasting in Crowds: A Deep Learning Perspective](https://mlsanity.com/paper/2007.03639/) (2020)
- [Peeking into the Future: Predicting Future Person Activities and Locations in Videos](https://mlsanity.com/paper/1902.03748/) (2019)
- [Core Challenges of Social Robot Navigation: A Survey](https://mlsanity.com/paper/2103.05668/) (2021)
- [Learning to See by Moving](https://mlsanity.com/paper/1505.01596/) (2015)
- [The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction](https://mlsanity.com/paper/1912.06445/) (2019)
- [Principles and Guidelines for Evaluating Social Robot Navigation Algorithms](https://mlsanity.com/paper/2306.16740/) (2023)
- [Auxiliary Tasks Speed Up Learning PointGoal Navigation](https://mlsanity.com/paper/2007.04561/) (2020)
- [Decentralized Structural-RNN for Robot Crowd Navigation with Deep Reinforcement Learning](https://mlsanity.com/paper/2011.04820/) (2020)
- [A Survey on Socially Aware Robot Navigation: Taxonomy and Future Challenges](https://mlsanity.com/paper/2311.06922/) (2023)
- [TH"OR: Human-Robot Navigation Data Collection and Accurate Motion Trajectories Dataset](https://mlsanity.com/paper/1909.04403/) (2019)
- [Robot Navigation in Constrained Pedestrian Environments using Reinforcement Learning](https://mlsanity.com/paper/2010.08600/) (2020)
- [A Survey on Human-aware Robot Navigation](https://mlsanity.com/paper/2106.11650/) (2021)
- [Risk-Sensitive Sequential Action Control with Multi-Modal Human Trajectory Forecasting for Safe Crowd-Robot Interaction](https://mlsanity.com/paper/2009.05702/) (2020)
- [Multi-Robot Cooperative Socially-Aware Navigation Using Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2309.15234/) (2023)
- [Deep-Learned Collision Avoidance Policy for Distributed Multi-Agent Navigation](https://mlsanity.com/paper/1609.06838/) (2016)

## Built on by (most cited first, 10 of 10)

- [SeeGround: See and Ground for Zero-Shot Open-Vocabulary 3D Visual Grounding](https://mlsanity.com/paper/2412.04383/) (2024)
- [TIC-VLA: A Think-in-Control Vision-Language-Action Model for Robot Navigation in Dynamic Environments](https://mlsanity.com/paper/2602.02459/) (2026)
- [View-on-Graph: Zero-shot 3D Visual Grounding via Vision-Language Reasoning on Scene Graphs](https://mlsanity.com/paper/2512.09215/) (2025)
- [Zero-Shot 3D Visual Grounding from Vision-Language Models](https://mlsanity.com/paper/2505.22429/) (2025)
- [FLUX: Accelerating Cross-Embodiment Generative Navigation Policies via Rectified Flow and Static-to-Dynamic Learning](https://mlsanity.com/paper/2603.12806/) (2026)
- [SEMNAV: Enhancing Visual Semantic Navigation in Robotics through Semantic Segmentation](https://mlsanity.com/paper/2506.01418/) (2025)
- [Learning to Navigate Socially Through Proactive Risk Perception](https://mlsanity.com/paper/2510.07871/) (2025)
- [BEACON: Language-Conditioned Navigation Affordance Prediction under Occlusion](https://mlsanity.com/paper/2603.09961/) (2026)
- [Think When It Matters: Conditional VLM Reasoning for Social Navigation with RL Policies](https://mlsanity.com/paper/2607.10991/) (2026)
- [DynActiveGS: Active Gaussian Splatting for Dynamic Scene Reconstruction](https://mlsanity.com/paper/2608.01178/) (2026)

## Abstract (arXiv)

To navigate safely and efficiently in crowded spaces, robots should not only perceive the current state of the environment but also anticipate future human movements. In this paper, we propose a reinforcement learning architecture, namely Falcon, to tackle socially-aware navigation by explicitly predicting human trajectories and penalizing actions that block future human paths. To facilitate realistic evaluation, we introduce a novel SocialNav benchmark containing two new datasets, Social-HM3D and Social-MP3D. This benchmark offers large-scale photo-realistic indoor scenes populated with a reasonable amount of human agents based on scene area size, incorporating natural human movements and trajectory patterns. We conduct a detailed experimental analysis with the state-of-the-art learning-based method and two classic rule-based path-planning algorithms on the new benchmark. The results demonstrate the importance of future prediction and our method achieves the best task success rate of 55% while maintaining about 90% personal space compliance. We will release our code and datasets. Videos of demonstrations can be viewed at https://zeying-gong.github.io/projects/falcon/ .