# Bayesian Inverse Games with High-Dimensional Multi-Modal Observations

arXiv: https://arxiv.org/abs/2601.00696 · HTML: https://mlsanity.com/paper/2601.00696/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 33 of its 56 references that are in the map.
- First posted on arXiv 2 Jan 2026.

## Builds on (most cited first, 30 of 33)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [DINOv3](https://mlsanity.com/paper/2508.10104/) (2025)
- [AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting](https://mlsanity.com/paper/2103.14023/) (2021)
- [DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents](https://mlsanity.com/paper/1704.04394/) (2017)
- [Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoders](https://mlsanity.com/paper/1812.01784/) (2018)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling](https://mlsanity.com/paper/2102.13156/) (2021)
- [Efficient Iterative Linear-Quadratic Approximations for Nonlinear Multi-Player General-Sum Differential Games](https://mlsanity.com/paper/1909.04694/) (2019)
- [Computational Rationalization: The Inverse Equilibrium Problem](https://mlsanity.com/paper/1308.3506/) (2013)
- [Maximum-Entropy Multi-Agent Dynamic Games: Forward and Inverse Solutions](https://mlsanity.com/paper/2110.01027/) (2021)
- [A Real-Time Game Theoretic Planner for Autonomous Two-Player Drone Racing](https://mlsanity.com/paper/1801.02302/) (2018)
- [Multimodal VAE Active Inference Controller](https://mlsanity.com/paper/2103.04412/) (2021)
- [ALGAMES: A Fast Augmented Lagrangian Solver for Constrained Dynamic Games](https://mlsanity.com/paper/2104.08452/) (2021)
- [VMLoc: Variational Fusion For Learning-Based Multimodal Camera Localization](https://mlsanity.com/paper/2003.07289/) (2020)
- [Learning Game-Theoretic Models of Multiagent Trajectories Using Implicit Layers](https://mlsanity.com/paper/2008.07303/) (2020)
- [Contingency Games for Multi-Agent Interaction](https://mlsanity.com/paper/2304.05483/) (2023)
- [Multimodal representation models for prediction and control from partial information](https://mlsanity.com/paper/1910.03854/) (2019)
- [Learning to Play Trajectory Games Against Opponents with Unknown Objectives](https://mlsanity.com/paper/2211.13779/) (2022)
- [Active Uncertainty Reduction for Human-Robot Interaction: An Implicit Dual Control Approach](https://mlsanity.com/paper/2202.07720/) (2022)
- [Energy-based Potential Games for Joint Motion Forecasting and Control](https://mlsanity.com/paper/2312.01811/) (2023)
- [The Computation of Approximate Generalized Feedback Nash Equilibria](https://mlsanity.com/paper/2101.02900/) (2021)
- [Cost Inference for Feedback Dynamic Games from Noisy Partial State Observations and Incomplete Trajectories](https://mlsanity.com/paper/2301.01398/) (2023)
- [Physics-informed Variational Autoencoders for Improved Robustness to Environmental Factors of Variation](https://mlsanity.com/paper/2210.10418/) (2022)
- [Think Deep and Fast: Learning Neural Nonlinear Opinion Dynamics from Inverse Dynamic Games for Split-Second Interactions](https://mlsanity.com/paper/2406.09810/) (2024)
- [Inverse Mixed Strategy Games with Generative Trajectory Models](https://mlsanity.com/paper/2502.03356/) (2025)
- [Online and Offline Learning of Player Objectives from Partial Observations in Dynamic Games](https://mlsanity.com/paper/2302.01999/) (2023)

## Abstract (arXiv)

Many multi-agent interaction scenarios can be naturally modeled as noncooperative games, where each agent's decisions depend on others' future actions. However, deploying game-theoretic planners for autonomous decision-making requires a specification of all agents' objectives. To circumvent this practical difficulty, recent work develops maximum likelihood techniques for solving inverse games that can identify unknown agent objectives from interaction data. Unfortunately, these methods only infer point estimates and do not quantify estimator uncertainty; correspondingly, downstream planning decisions can overconfidently commit to unsafe actions. We present an approximate Bayesian inference approach for solving the inverse game problem, which can incorporate observation data from multiple modalities and be used to generate samples from the Bayesian posterior over the hidden agent objectives given limited sensor observations in real time. Concretely, the proposed Bayesian inverse game framework trains a structured variational autoencoder with an embedded differentiable Nash game solver on interaction datasets and does not require labels of agents' true objectives. Extensive experiments show that our framework successfully learns prior and posterior distributions, improves inference quality over maximum likelihood estimation-based inverse game approaches, and enables safer downstream decision-making without sacrificing efficiency. When trajectory information is uninformative or unavailable, multimodal inference further reduces uncertainty by exploiting additional observation modalities.