mlsanityData to 7 Oct 2026

RoboPack: Learning Tactile-Informed Dynamics Models for Dense Packing

arXiv 2407.01418 ↗First posted 1 Jul 2024cs.RO cs.AI cs.LGcited by 18

Not on the map. Why?

Abstract (arXiv)

Tactile feedback is critical for understanding the dynamics of both rigid and deformable objects in many manipulation tasks, such as non-prehensile manipulation and dense packing. We introduce an approach that combines visual and tactile sensing for robotic manipulation by learning a neural, tactile-informed dynamics model. Our proposed framework, RoboPack, employs a recurrent graph neural network to estimate object states, including particles and object-level latent physics information, from historical visuo-tactile observations and to perform future state predictions. Our tactile-informed dynamics model, learned from real-world data, can solve downstream robotics tasks with model-predictive control. We demonstrate our approach on a real robot equipped with a compliant Soft-Bubble tactile sensor on non-prehensile manipulation and dense packing tasks, where the robot must infer the physics properties of objects from direct and indirect interactions. Trained on only an average of 30 minutes of real-world interaction data per task, our model can perform online adaptation and make touch-informed predictions. Through extensive evaluations in both long-horizon dynamics prediction and real-world manipulation, our method demonstrates superior effectiveness compared to previous learning-based and physics-based simulation systems.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 47 of its 67 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

47 of 67 references.

References · 47

  1. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2020Learning to Simulate Complex Physics with Graph Networkscited by 867
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  8. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  9. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  10. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  11. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  12. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
35 more
  1. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 220
  2. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  3. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 200
  4. 2020DIGIT: A Novel Design for a Low-Cost Compact High-Resolution Tactile Sensor with Application to In-Hand Manipulationcited by 164
  5. 2018Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guaranteescited by 138
  6. 2022VIOLA: Imitation Learning for Vision-Based Manipulation with Object Proposal Priorscited by 135
  7. 2020Learning to Control PDEs with Differentiable Physicscited by 131
  8. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  9. 2022RAMBO-RL: Robust Adversarial Model-Based Offline Reinforcement Learningcited by 115
  10. 20213D Neural Scene Representations for Visuomotor Controlcited by 110
  11. 2018More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touchcited by 100
  12. 2020Offline Reinforcement Learning from Images with Latent Space Modelscited by 94
  13. 2023General In-Hand Object Rotation with Vision and Touchcited by 87
  14. 2020Causal Discovery in Physical Systems from Videoscited by 77
  15. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 75
  16. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  17. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 69
  18. 2021Learning Visible Connectivity Dynamics for Cloth Smoothingcited by 61
  19. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  20. 2023Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensingcited by 51
  21. 2018GelSlim: A High-Resolution, Compact, Robust, and Calibrated Tactile-sensing Fingercited by 50
  22. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  23. 2023D3Fields: Dynamic 3D Descriptor Fields for Zero-Shot Generalizable Rearrangementcited by 35
  24. 2019Manipulation by Feel: Touch-Based Control with Deep Predictive Modelscited by 34
  25. 2021Constructing Neural Network-Based Models for Simulating Dynamical Systemscited by 29
  26. 2023Neural feels with neural fields: Visuo-tactile perception for in-hand manipulationcited by 28
  27. 2023Dynamic-Resolution Model Learning for Object Pile Manipulationcited by 22
  28. 2020Soft-Bubble grippers for robust and perceptive manipulationcited by 18
  29. 20239DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimationcited by 16
  30. 2023Predicting Object Interactions with Behavior Primitives: An Application in Stowing Taskscited by 12
  31. 2023A Survey on Offline Model-Based Reinforcement Learningcited by 7
  32. 2021Deformable Elasto-Plastic Object Shaping using an Elastic Hand and Model-Based Reinforcement Learningcited by 6
  33. 2021Deep Visual Navigation under Partial Observabilitycited by 6
  34. 2020Model-Based Manipulation of Linear Flexible Objects with Visual Curvature Feedbackcited by 3
  35. 2021SEED: Series Elastic End Effectors in 6D for Visuotactile Tool Usecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 10; 2026: 8.

18 papers.

Sort:

By area:

  1. 2025Reactive Diffusion Policy: Slow-Fast Visual-Tactile Policy Learning for Contact-Rich Manipulationcited by 103
  2. 2025VLA-Touch: Enhancing Vision-Language-Action Models with Dual-Level Tactile Feedbackcited by 37
  3. 2025Do Vision-Language Models Have Internal World Models? Towards an Atomic Evaluationcited by 23
  4. 2025Particle-Grid Neural Dynamics for Learning Deformable Object Models from RGB-D Videoscited by 22
  5. 2025Diffusion Dynamics Models with Generative State Estimation for Cloth Manipulationcited by 16
  6. 2025VT-Refine: Learning Bimanual Assembly with Visuo-Tactile Feedback via Simulation Fine-Tuningcited by 10
  7. 2025Enhancing Tactile-based Reinforcement Learning for Robotic Controlcited by 5
  8. 2026Graph World Models: Concepts, Taxonomy, and Future Directionscited by 5
  9. 2026UniTacVLA: Unified Tactile Understanding and Prediction in Vision Language Action Modelscited by 4
  10. 2026In-Context Robot Learning with VLM Agentscited by 4
  11. 2025X-Capture: An Open-Source Portable Device for Multi-Sensory Learningcited by 3
  12. 2026TacWAM: Anchor-Guided World Action Model with Mechanics-Aware Tactile Predictioncited by 3
6 more
  1. 2025Localized Graph-Based Neural Dynamics Models for Terrain Manipulationcited by 1
  2. 2026PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamicscited by 1
  3. 2026Rapid Embodiment Adaptation for Quadrupedal Locomotioncited by 1
  4. 2025ControlTac: Scaling Tactile Data with Physically Controlled Tactile Image Generationcited by 0
  5. 2026Velox: Learning Representations of 4D Geometry and Appearancecited by 0
  6. 2026CoStream: Composing Simple Behaviors for Generalizable Complex Manipulationcited by 0