mlsanityData to 14 Oct 2026

Modeling Fly Social Behavior with Agent-Centric Pose Forecasting

arXiv 2607.19548 ↗First posted 21 Jul 2026cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal representations, and how this maps to action -- remains a central challenge in neuroethology. We explore how well large generative models capture properties of fly social behavior. We combine transformers with agent-centric representations. Models input egocentric sensory observations and output egocentric movements, mirroring biological constraints. Social behavior emerges from agents independently sensing and responding to one another. We demonstrate that this simple model captures many properties of fly behavior, outperforming the world-centric representation and achieving SOTA on MABe 2022-fly. We systematically evaluate properties of the representation important for capturing the behavior distribution through a series of lesioning experiments. These comparisons were enabled by a new, general-purpose library for animal-centric pose forecasting.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

23 of 36 references.

References · 23

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  4. 2018Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networkscited by 690
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  6. 2017On human motion prediction using recurrent neural networkscited by 338
  7. 2021AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecastingcited by 331
  8. 2017DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agentscited by 310
  9. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 246
  10. 2019Human Motion Trajectory Prediction: A Surveycited by 232
  11. 2022Multi-Agent Reinforcement Learning is a Sequence Modeling Problemcited by 155
  12. 2018QuaterNet: A Quaternion-based Recurrent Model for Human Motioncited by 109
11 more
  1. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  2. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  3. 2021TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wildcited by 52
  4. 2021Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approachescited by 30
  5. 2022SoMoFormer: Multi-Person Pose Forecasting with Transformerscited by 23
  6. 2022H3WB: Human3.6M 3D WholeBody Dataset and Benchmarkcited by 23
  7. 2016Learning recurrent representations for hierarchical behavior modelingcited by 16
  8. 2023Vision-based Multi-future Trajectory Prediction: A Surveycited by 8
  9. 2023Best Practices for 2-Body Pose Forecastingcited by 8
  10. 2015Recurrent Network Models for Human Dynamicscited by 4
  11. 2020Evaluation metrics for behaviour modelingcited by 1

Built on by

Not cited yet.