# Data-driven simulator of multi-animal behavior with unknown dynamics via offline and online reinforcement learning

arXiv: https://arxiv.org/abs/2510.10451 · HTML: https://mlsanity.com/paper/2510.10451/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 19 of its 44 references that are in the map.
- First posted on arXiv 12 Oct 2025.

## Builds on (most cited first, 19 of 19)

- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Prioritized Experience Replay](https://mlsanity.com/paper/1511.05952/) (2015)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [Learning Representations for Counterfactual Inference](https://mlsanity.com/paper/1605.03661/) (2016)
- [Learning Agile Robotic Locomotion Skills by Imitating Animals](https://mlsanity.com/paper/2004.00784/) (2020)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [Sim-to-Real Robot Learning from Pixels with Progressive Nets](https://mlsanity.com/paper/1610.04286/) (2016)
- [Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learning](https://mlsanity.com/paper/1703.02949/) (2017)
- [Cross-Domain Imitation Learning via Optimal Transport](https://mlsanity.com/paper/2110.03684/) (2021)
- [Imitation Learning with Sinkhorn Distances](https://mlsanity.com/paper/2008.09167/) (2020)
- [Learning recurrent representations for hierarchical behavior modeling](https://mlsanity.com/paper/1611.00094/) (2016)
- [Estimating counterfactual treatment outcomes over time in complex multiagent scenarios](https://mlsanity.com/paper/2206.01900/) (2022)
- [Decentralized policy learning with partial observation and mechanical constraints for multiperson modeling](https://mlsanity.com/paper/2007.03155/) (2020)
- [Action valuation of on- and off-ball soccer players based on multi-agent deep reinforcement learning](https://mlsanity.com/paper/2305.17886/) (2023)
- [Adaptive action supervision in reinforcement learning from real-world multi-agent demonstrations](https://mlsanity.com/paper/2305.13030/) (2023)

## Abstract (arXiv)

Simulators of animal movements play a valuable role in studying behavior. Advances in imitation learning for robotics have expanded possibilities for reproducing human and animal movements. A key challenge for realistic multi-animal simulation in biology is bridging the gap between unknown real-world transition models and their simulated counterparts. Because locomotion dynamics are seldom known, relying solely on mathematical models is insufficient; constructing a simulator that both reproduces real trajectories and supports reward-driven optimization remains an open problem. We introduce a data-driven simulator for multi-animal behavior based on deep reinforcement learning and counterfactual simulation. We address the ill-posed nature of the problem caused by high degrees of freedom in locomotion by estimating movement variables of an incomplete transition model as actions within an RL framework. We also employ a distance-based pseudo-reward to align and compare states between cyber and physical spaces. Validated on artificial agents, flies, newts, and silkmoth, our approach achieves higher reproducibility of species-specific behaviors and improved reward acquisition compared with standard imitation and RL methods. Moreover, it enables counterfactual behavior prediction in novel experimental settings and supports multi-individual modeling for flexible what-if trajectory generation, suggesting its potential to simulate and elucidate complex multi-animal behaviors.