Agent-to-Sim: Learning Interactive Behavior Models from Casual Longitudinal Videos
Not on the map. Why?
Abstract (arXiv)
We present Agent-to-Sim (ATS), a framework for learning interactive behavior models of 3D agents from casual longitudinal video collections. Different from prior works that rely on marker-based tracking and multiview cameras, ATS learns natural behaviors of animal and human agents non-invasively through video observations recorded over a long time-span (e.g., a month) in a single environment. Modeling 3D behavior of an agent requires persistent 3D tracking (e.g., knowing which point corresponds to which) over a long time period. To obtain such data, we develop a coarse-to-fine registration method that tracks the agent and the camera over time through a canonical 3D space, resulting in a complete and persistent spacetime 4D representation. We then train a generative model of agent behaviors using paired data of perception and motion of an agent queried from the 4D reconstruction. ATS enables real-to-sim transfer from video recordings of an agent to an interactive behavior simulator. We demonstrate results on pets (e.g., cat, dog, bunny) and human given monocular RGBD videos captured by a smartphone.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 59 of its 83 references that are in the map. First posted on arXiv 21 Oct 2024.
Builds on
59 of 83 references.
References · 59
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
- 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,141
- 2022Classifier-Free Diffusion Guidancecited by 5,717
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
- 2023DUSt3R: Geometric 3D Vision Made Easycited by 1,526
- 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
- 2020Nerfies: Deformable Neural Radiance Fieldscited by 1,194
- 2022Human Motion Diffusion Modelcited by 1,062
38 more
- 2020GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fieldscited by 784
- 2023Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesiscited by 731
- 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
- 2018From Coarse to Fine: Robust Hierarchical Localization at Large Scalecited by 556
- 2021Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Datasetcited by 532
- 2019MultiPath: Multiple Probabilistic Anchor Trajectory Hypotheses for Behavior Predictioncited by 417
- 2022Decomposing NeRF for Editing via Feature Field Distillationcited by 370
- 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
- 2023Human Motion Diffusion as a Generative Priorcited by 354
- 2023Humans in 4D: Reconstructing and Tracking Humans with Transformerscited by 344
- 2023ReconFusion: 3D Reconstruction with Diffusion Priorscited by 317
- 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
- 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
- 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 220
- 2023MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusioncited by 210
- 2021Stochastic Scene-Aware Motion Predictioncited by 203
- 2022Monocular Dynamic View Synthesis: A Reality Checkcited by 199
- 2021BANMo: Building Animatable 3D Neural Models from Many Casual Videoscited by 195
- 2020Long-term Human Motion Prediction with Scene Contextcited by 191
- 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
- 2021Character Controllers Using Motion VAEscited by 182
- 2021SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networkscited by 181
- 2019PRECOG: PREdiction Conditioned On Goals in Visual Multi-Agent Settingscited by 178
- 2024Probing the 3D Awareness of Visual Foundation Modelscited by 175
- 2016Panoptic Studio: A Massively Multiview System for Social Interaction Capturecited by 173
- 2020From Goals, Waypoints & Paths To Long Term Human Trajectory Forecastingcited by 166
- 2021BEHAVIOR: Benchmark for Everyday Household Activities in Virtual, Interactive, and Ecological Environmentscited by 161
- 2023MotionLM: Multi-Agent Motion Forecasting as Language Modelingcited by 154
- 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
- 2023Decoupling Human and Camera Motion from Videos in the Wildcited by 140
- 2024BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulationcited by 140
- 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusioncited by 137
- 2019Neural-Guided RANSAC: Learning Where to Sample Model Hypothesescited by 129
- 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
- 2023Accelerated Coordinate Encoding: Learning to Relocalize in Minutes using RGB and Posescited by 107
- 2023Synthesizing Physical Character-Scene Interactionscited by 102
- 2021DOVE: Learning Deformable 3D Objects by Watching Videoscited by 61
- 2021LOKI: Long Term and Key Intentions for Trajectory Predictioncited by 57
50 most cited of 59 All 59 (JSON)
Built on by
Citing papers per year: 2025: 3.
3 papers.