# NeuROK: Generative 4D Neural Object Kinematics

arXiv: https://arxiv.org/abs/2605.30347 · HTML: https://mlsanity.com/paper/2605.30347/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 92 of its 120 references that are in the map.
- First posted on arXiv 28 May 2026.

## Builds on (most cited first, 30 of 92)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [VBench: Comprehensive Benchmark Suite for Video Generative Models](https://mlsanity.com/paper/2311.17982/) (2023)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [A Point Set Generation Network for 3D Object Reconstruction from a Single Image](https://mlsanity.com/paper/1612.00603/) (2016)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Learning Mesh-Based Simulation with Graph Networks](https://mlsanity.com/paper/2010.03409/) (2020)
- [Objaverse-XL: A Universe of 10M+ 3D Objects](https://mlsanity.com/paper/2307.05663/) (2023)
- [SAPIEN: A SimulAted Part-based Interactive ENvironment](https://mlsanity.com/paper/2003.08515/) (2020)
- [Interaction Networks for Learning about Objects, Relations and Physics](https://mlsanity.com/paper/1612.00222/) (2016)
- [3DShape2VecSet: A 3D Shape Representation for Neural Fields and Generative Diffusion Models](https://mlsanity.com/paper/2301.11445/) (2023)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamics](https://mlsanity.com/paper/2311.12198/) (2023)
- [3D Menagerie: Modeling the 3D shape and pose of animals](https://mlsanity.com/paper/1611.07700/) (2016)
- [Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation](https://mlsanity.com/paper/2306.17115/) (2023)
- [Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluids](https://mlsanity.com/paper/1810.01566/) (2018)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [PhysDreamer: Physics-Based Interaction with 3D Objects via Video Generation](https://mlsanity.com/paper/2404.13026/) (2024)
- [Ditto: Building Digital Twins of Articulated Objects from Interaction](https://mlsanity.com/paper/2202.08227/) (2022)
- [WorldScore: A Unified Evaluation Benchmark for World Generation](https://mlsanity.com/paper/2504.00983/) (2025)
- [PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identification](https://mlsanity.com/paper/2303.05512/) (2023)
- [A Compositional Object-Based Approach to Learning Physical Dynamics](https://mlsanity.com/paper/1612.00341/) (2016)
- [PARIS: Part-level Reconstruction and Motion Analysis for Articulated Objects](https://mlsanity.com/paper/2308.07391/) (2023)
- [3D Neural Scene Representations for Visuomotor Control](https://mlsanity.com/paper/2107.04004/) (2021)
- [URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images](https://mlsanity.com/paper/2405.11656/) (2024)
- [Neural Cages for Detail-Preserving 3D Deformations](https://mlsanity.com/paper/1912.06395/) (2019)
- [Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints](https://mlsanity.com/paper/2010.13581/) (2020)
- [Reconstruction and Simulation of Elastic Objects with Spring-Mass 3D Gaussians](https://mlsanity.com/paper/2403.09434/) (2024)

## Built on by (most cited first, 3 of 3)

- [Visual General Intelligence: A White Paper](https://mlsanity.com/paper/2608.25924/) (2026)
- [LaGSplat: Inferring Physics-Governed Interactive Simulation from Monocular Video Using Latent Lagrangian Gaussian Splatting](https://mlsanity.com/paper/2608.16324/) (2026)
- [PhysLDM: Latent Diffusion for High-Fidelity Deformable Simulation](https://mlsanity.com/paper/2610.07609/) (2026)

## Abstract (arXiv)

Data-driven approaches have revolutionized 3D vision, enabling transformers to effectively reconstruct and generate static 3D objects. However, generating simulative 4D dynamics -- realistic temporal deformations of static objects under various physical conditions -- remains challenging and often ad hoc, despite its importance in building comprehensive 3D world models. Most existing methods assume a predefined physical model and use system identification to estimate parameters, restricting these methods to specific categories and small-scale datasets. We propose that these restrictions can be overcome by learning a data-driven kinematic state parameterization for object-centric physical systems. Specifically, we learn both a latent space representing all possible states of the object and a decoder that maps any sampled latent to a plausibly deformed shape of the object. We refer to this parameterization as Neural Object Kinematics (NeuROK), and learn a transformer-based encoder-decoder model on a curated large-scale 4D dataset. This formulation and the learned model significantly simplify the generation of simulative dynamics since we only need to consider the dynamics within a low-dimensional latent space from the Lagrangian mechanics' perspective in classical physics. We demonstrate the effectiveness and generality of this neural simulation framework across diverse dynamic object types, showing clear advantages over prior works. Project page: https://chen-geng.com/neurok