mlsanityData to 14 Oct 2026

Modeling the Real World with High-Density Visual Particle Dynamics

arXiv 2406.19800 ↗First posted 28 Jun 2024cs.LG cs.ROcited by 12

On the map: World models → Latent predictive world models

Abstract (arXiv)

We present High-Density Visual Particle Dynamics (HD-VPD), a learned world model that can emulate the physical dynamics of real scenes by processing massive latent point clouds containing 100K+ particles. To enable efficiency at this scale, we introduce a novel family of Point Cloud Transformers (PCTs) called Interlacers leveraging intertwined linear-attention Performer layers and graph-based neighbour attention layers. We demonstrate the capabilities of HD-VPD by modeling the dynamics of high degree-of-freedom bi-manual robots with two RGB-D cameras. Compared to the previous graph neural network approach, our Interlacer dynamics is twice as fast with the same prediction quality, and can achieve higher quality using 4x as many particles. We illustrate how HD-VPD can evaluate motion plan quality with robotic box pushing and can grasping tasks. See videos and particle dynamics rendered by HD-VPD at https://sites.google.com/view/hd-vpd.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Latent predictive world models, one of 11 directions in World models. Builds on 25 of its 29 references that are in the map. First posted on arXiv 28 Jun 2024.

Builds on

25 of 29 references.

References · 25

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  2. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  3. 2018World Modelscited by 1,513
  4. 2020Point Transformercited by 1,470
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 998
  6. 2020Learning to Simulate Complex Physics with Graph Networkscited by 880
  7. 2020Learning Mesh-Based Simulation with Graph Networkscited by 868
  8. 2020PCT: Point cloud transformercited by 732
  9. 2019RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Cloudscited by 585
  10. 2017AirSim: High-Fidelity Visual and Physical Simulation for Autonomous Vehiclescited by 523
  11. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 509
  12. 2022Point-NeRF: Point-based Neural Radiance Fieldscited by 503
13 more
  1. 2023Learning Interactive Real-World Simulatorscited by 458
  2. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  3. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 220
  4. 20213D Neural Scene Representations for Visuomotor Controlcited by 111
  5. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  6. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  7. 2022Learning rigid dynamics with face interaction graph networkscited by 43
  8. 2023SARA-RT: Scaling up Robotics Transformers with Self-Adaptive Robust Attentioncited by 20
  9. 2024Embodied AI with Two Arms: Zero-shot Learning, Safety and Modularitycited by 19
  10. 2023Learning 3D Particle-based Simulators from RGB-D Videoscited by 17
  11. 20233D-IntPhys: Towards More Generalized 3D-grounded Visual Intuitive Physics under Challenging Scenescited by 12
  12. 2024Learning rigid-body simulators over implicit shapes for large-scale scenes and visioncited by 12
  13. 2023Hierarchical Graph Neural Networks for Particle Track Reconstructioncited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 9.

12 papers.

  1. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  2. 2025Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videoscited by 25
  3. 2025Learning 3D-Gaussian Simulators from RGB Videoscited by 8
  4. 2024Linear Transformer Topological Masking with Graph Random Featurescited by 7
  5. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 6
  6. 2026SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulationcited by 4
  7. 2026RigidFormer: Learning Rigid Dynamics using Transformerscited by 3
  8. 2026Learning Physics-Guided Residual Dynamics for Deformable Object Simulationcited by 2
  9. 2026PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamicscited by 2
  10. 20263D Point World Models: Point Completion Enables More Accurate Dynamics Learningcited by 1
  11. 2026PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamicscited by 1
  12. 2026Point Cloud Sequence Encoding for Material-conditioned Graph Network Simulatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation
  2. 2026Learning a Particle Dynamics Model with Real-world Videos
  3. 2026AcrossVAM1.0: Particle World Modeling for Text-Assisted Robot Video Prediction
  4. 2024Physically Embodied Gaussian Splatting: A Realtime Correctable World Model for Robotics
  5. 2025FlowDreamer: A RGB-D World Model with Flow-based Motion Representations for Robot Manipulation
  6. 2026Latent Particle World Models: Self-supervised Object-centric Stochastic Dynamics Modeling
  7. 2026RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation
  8. 2026ParticleSplat: Self-supervised Object-centric Latent Particle Splatting