mlsanityData to 7 Oct 2026

Learning to Poke by Poking: Experiential Learning of Intuitive Physics

arXiv 1606.07419 ↗First posted 23 Jun 2016cs.CV cs.AI cs.ROcited by 152

On the map: World models → Embodied and robot world models

Abstract (arXiv)

We investigate an experiential learning paradigm for acquiring an internal model of intuitive physics. Our model is evaluated on a real-world robotic manipulation task that requires displacing objects to target locations by poking. The robot gathered over 400 hours of experience by executing more than 100K pokes on different objects. We propose a novel approach based on deep neural networks for modeling the dynamics of robot's interactions directly from images, by jointly estimating forward and inverse models of dynamics. The inverse model objective provides supervision to construct informative visual features, which the forward model can then predict and in turn regularize the feature space for the inverse model. The interplay between these two objectives creates useful, accurate models that can then be used for multi-step decision making. This formulation has the additional benefit that it is possible to learn forward models in an abstract feature space and thus alleviate the need of predicting pixels. Our experiments show that this joint modeling approach outperforms alternative methods.

Key facts. Cited by 152 papers in the map (data through 7 Oct 2026). Placed in Embodied and robot world models, one of 11 directions in World models. Builds on 0 of its 35 references that are in the map. First posted on arXiv 23 Jun 2016.

Builds on

0 of 35 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2018: 3; 2019: 0; 2020: 4; 2021: 44; 2022: 28; 2023: 19; 2024: 14; 2025: 20; 2026: 20.

152 papers.

Sort:

By area:

  1. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  2. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 748
  3. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
  4. 2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
  5. 2022Is Conditional Generative Modeling all you need for Decision-Making?cited by 504
  6. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  7. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  8. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 249
  9. 2022Masked Visual Pre-training for Motor Controlcited by 246
  10. 2022Real-World Robot Learning with Masked Visual Pre-trainingcited by 241
  11. 2018Model-Ensemble Trust-Region Policy Optimizationcited by 214
  12. 2021Reinforcement Learning with Prototypical Representationscited by 179
38 more
  1. 2024Predictive Inverse Dynamics Models are Scalable Learners for Robotic Manipulationcited by 159
  2. 2025Do generative video models understand physical principles?cited by 151
  3. 2021Physion: Evaluating Physical Prediction from Vision in Humans and Machinescited by 145
  4. 2022Towards Human-Level Bimanual Dexterous Manipulation with Reinforcement Learningcited by 113
  5. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  6. 2021gradSim: Differentiable simulation for system identification and visuomotor controlcited by 100
  7. 2023Compositional Foundation Models for Hierarchical Planningcited by 92
  8. 2020State-Only Imitation Learning for Dexterous Manipulationcited by 90
  9. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  10. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 74
  11. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 72
  12. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  13. 2025A Comprehensive Survey on World Models for Embodied AIcited by 65
  14. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  15. 2022Learning Multi-Object Dynamics with Compositional Neural Radiance Fieldscited by 63
  16. 2021Learning Task Informed Abstractionscited by 58
  17. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  18. 2021AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactionscited by 56
  19. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  20. 2023Physion++: Evaluating Physical Scene Understanding that Requires Online Inference of Different Physical Propertiescited by 53
  21. 2024AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulationcited by 52
  22. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  23. 2021Towards Robust Bisimulation Metric Learningcited by 49
  24. 2024RoboEXP: Action-Conditioned Scene Graph via Interactive Exploration for Robotic Manipulationcited by 48
  25. 2021O2O-Afford: Annotation-Free Large-Scale Object-Object Affordance Learningcited by 46
  26. 2022Vision-Based Manipulators Need to Also See from Their Handscited by 46
  27. 2021Curious Representation Learning for Embodied Intelligencecited by 43
  28. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
  29. 2023Paxion: Patching Action Knowledge in Video-Language Foundation Modelscited by 42
  30. 2022Learning Physical Dynamics with Subequivariant Graph Neural Networkscited by 38
  31. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  32. 2023See to Touch: Learning Tactile Dexterity through Visual Incentivescited by 33
  33. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  34. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  35. 2021Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulationcited by 30
  36. 2025Whole-Body Conditioned Egocentric Video Predictioncited by 30
  37. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  38. 2021Which Mutual-Information Representation Learning Objectives are Sufficient for Control?cited by 28

50 most cited of 152 All 152 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2019Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresight
  2. 2025Learning Sequential Kinematic Models from Demonstrations for Multi-Jointed Articulated Objects
  3. 2026PokeVLA: Empowering Pocket-Sized Vision-Language-Action Model with Comprehensive World Knowledge Guidance
  4. 2020Model-Based Inverse Reinforcement Learning from Visual Demonstrations
  5. 2023Scaling Robot Learning with Semantically Imagined Experience
  6. 2025RoboScape: Physics-informed Embodied World Model
  7. 2026World Models for Robotic Manipulation: A Survey
  8. 2026EMPIRIC: Experiment-Driven Learning of Residual World Models for Robot Planning