# Multi-Agent Variational Occlusion Inference Using People as Sensors

arXiv: https://arxiv.org/abs/2109.02173 · HTML: https://mlsanity.com/paper/2109.02173/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 51 references that are in the map.
- First posted on arXiv 5 Sep 2021.

## Builds on (most cited first, 22 of 22)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [nuScenes: A multimodal dataset for autonomous driving](https://mlsanity.com/paper/1903.11027/) (2019)
- [Categorical Reparameterization with Gumbel-Softmax](https://mlsanity.com/paper/1611.01144/) (2016)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Generating Sentences from a Continuous Space](https://mlsanity.com/paper/1511.06349/) (2015)
- [INTERACTION Dataset: An INTERnational, Adversarial and Cooperative moTION Dataset in Interactive Driving Scenarios with Semantic Maps](https://mlsanity.com/paper/1910.03088/) (2019)
- [Self-Supervised Visual Planning with Temporal Skip Connections](https://mlsanity.com/paper/1710.05268/) (2017)
- [Multimodal Probabilistic Model-Based Planning for Human-Robot Interaction](https://mlsanity.com/paper/1710.09483/) (2017)
- [Learning to Look around Objects for Top-View Representations of Outdoor Scenes](https://mlsanity.com/paper/1803.10870/) (2018)
- [Joint Attention in Autonomous Driving (JAAD)](https://mlsanity.com/paper/1609.04741/) (2016)
- [Dynamic Environment Prediction in Urban Scenes using Recurrent Representation Learning](https://mlsanity.com/paper/1904.12374/) (2019)
- [Evidential Sparsification of Multimodal Latent Spaces in Conditional Variational Autoencoders](https://mlsanity.com/paper/2010.09164/) (2020)
- [Seeing Behind Things: Extending Semantic Segmentation to Occluded Regions](https://mlsanity.com/paper/1906.02885/) (2019)
- [ROLL: Visual Self-Supervised Reinforcement Learning with Object Reasoning](https://mlsanity.com/paper/2011.06777/) (2020)
- [A Taxonomy and Review of Algorithms for Modeling and Predicting Human Driver Behavior](https://mlsanity.com/paper/2006.08832/) (2020)
- [People as Sensors: Imputing Maps from Human Actions](https://mlsanity.com/paper/1711.01022/) (2017)
- [Planning Paths Through Unknown Space by Imagining What Lies Therein](https://mlsanity.com/paper/2011.07316/) (2020)
- [Behavior Planning of Autonomous Cars with Social Perception](https://mlsanity.com/paper/1905.00988/) (2019)
- [Semantic Foreground Inpainting from Weak Supervision](https://mlsanity.com/paper/1909.04564/) (2019)
- [Dynamic Hilbert Maps: Real-Time Occupancy Predictions in Changing Environment](https://mlsanity.com/paper/1912.02149/) (2019)
- [HMPO: Human Motion Prediction in Occluded Environments for Safe Motion Planning](https://mlsanity.com/paper/2006.00424/) (2020)

## Built on by (most cited first, 10 of 10)

- [ReasonNet: End-to-End Driving with Temporal and Global Reasoning](https://mlsanity.com/paper/2305.10507/) (2023)
- [Interpretable Self-Aware Neural Networks for Robust Trajectory Prediction](https://mlsanity.com/paper/2211.08701/) (2022)
- [Scene Informer: Anchor-based Occlusion Inference and Trajectory Prediction in Partially Observable Environments](https://mlsanity.com/paper/2309.13893/) (2023)
- [Occlusion-Aware Crowd Navigation Using People as Sensors](https://mlsanity.com/paper/2210.00552/) (2022)
- [Self-supervised Multi-future Occupancy Forecasting for Autonomous Driving](https://mlsanity.com/paper/2407.21126/) (2024)
- [RMP: A Random Mask Pretrain Framework for Motion Prediction](https://mlsanity.com/paper/2309.08989/) (2023)
- [Planning with Occluded Traffic Agents using Bi-Level Variational Occlusion Models](https://mlsanity.com/paper/2210.14584/) (2022)
- [LOPR: Latent Occupancy PRediction using Generative Models](https://mlsanity.com/paper/2210.01249/) (2022)
- [Social Occlusion Inference with Vectorized Representation for Autonomous Driving](https://mlsanity.com/paper/2303.10385/) (2023)
- [BlindSpotNet: Seeing Where We Cannot See](https://mlsanity.com/paper/2207.03870/) (2022)

## Abstract (arXiv)

Autonomous vehicles must reason about spatial occlusions in urban environments to ensure safety without being overly cautious. Prior work explored occlusion inference from observed social behaviors of road agents, hence treating people as sensors. Inferring occupancy from agent behaviors is an inherently multimodal problem; a driver may behave similarly for different occupancy patterns ahead of them (e.g., a driver may move at constant speed in traffic or on an open road). Past work, however, does not account for this multimodality, thus neglecting to model this source of aleatoric uncertainty in the relationship between driver behaviors and their environment. We propose an occlusion inference method that characterizes observed behaviors of human agents as sensor measurements, and fuses them with those from a standard sensor suite. To capture the aleatoric uncertainty, we train a conditional variational autoencoder with a discrete latent space to learn a multimodal mapping from observed driver trajectories to an occupancy grid representation of the view ahead of the driver. Our method handles multi-agent scenarios, combining measurements from multiple observed drivers using evidential theory to solve the sensor fusion problem. Our approach is validated on a cluttered, real-world intersection, outperforming baselines and demonstrating real-time capable performance. Our code is available at https://github.com/sisl/MultiAgentVariationalOcclusionInference .