# Structural Inference under Hidden Agents

arXiv: https://arxiv.org/abs/2609.18045 · HTML: https://mlsanity.com/paper/2609.18045/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 16 Sep 2026.

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation](https://mlsanity.com/paper/2107.03502/) (2021)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [BRITS: Bidirectional Recurrent Imputation for Time Series](https://mlsanity.com/paper/1805.10572/) (2018)
- [Learning Discrete Structures for Graph Neural Networks](https://mlsanity.com/paper/1903.11960/) (2019)
- [GP-VAE: Deep Probabilistic Time Series Imputation](https://mlsanity.com/paper/1907.04155/) (2019)
- [SAITS: Self-Attention-based Imputation for Time Series](https://mlsanity.com/paper/2202.08516/) (2022)
- [Conditional Neural Relational Inference for Interacting Systems](https://mlsanity.com/paper/2106.11083/) (2021)
- [A General Deep Learning Framework for Network Reconstruction and Dynamics Learning](https://mlsanity.com/paper/1812.11482/) (2018)
- [Neural Relational Inference with Fast Modular Meta-learning](https://mlsanity.com/paper/2310.07015/) (2023)
- [Neural Relational Inference with Efficient Message Passing Mechanisms](https://mlsanity.com/paper/2101.09486/) (2021)
- [Collective Relational Inference for learning heterogeneous interactions](https://mlsanity.com/paper/2305.00557/) (2023)
- [Learning Continuous Network Emerging Dynamics from Scarce Observations via Data-Adaptive Stochastic Processes](https://mlsanity.com/paper/2310.16466/) (2023)
- [Diffusion model for relational inference](https://mlsanity.com/paper/2401.16755/) (2024)

## Abstract (arXiv)

Recovering latent interaction structures from multi-agent dynamics is important for understanding and predicting interacting systems. Trajectory-based structural inference has achieved promising performance, but conventional formulations assume that the trajectories of all modeled agents are available. In practice, agents may become unobserved at deployment because of limited sensing, occlusion, or communication failure. Existing studies have considered unseen-node estimation, structural inference under partial observations, and missing-value imputation, yet the joint recovery of hidden-agent trajectories and their interactions remains underexplored. We formulate this problem as structural inference under hidden agents. Its key difficulty is a circular dependency: recovering interactions involving a hidden agent requires an estimate of its trajectory, while trajectory reconstruction can itself benefit from structural information. To address this challenge, we propose Structural Inference under Hidden Agents (SIHA), which combines structure-agnostic initialization with structure-guided iterative refinement. SIHA reconstructs hidden trajectories from visible observations, infers interactions using Neural Relational Inference, and feeds the estimated structure back into hidden-state reconstruction through multi-strength structural attention and iterative state--structure updates. Experiments on three benchmark dynamical systems demonstrate consistent improvements in visible-to-visible structural inference, while also showing benefits in hidden-state reconstruction and future prediction. Motion-capture experiments with simulated whole-limb occlusion further demonstrate its effectiveness in realistic hidden-agent settings.