mlsanityData to 13 Oct 2026

IMPASTO: Integrating Model-Based Planning with Learned Dynamics Models for Robotic Oil Painting Reproduction

arXiv 2603.29315 ↗First posted 31 Mar 2026cs.RO cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Robotic reproduction of oil paintings using soft brushes and pigments requires force-sensitive control of deformable tools, prediction of brushstroke effects, and multi-step stroke planning, often without human step-by-step demonstrations or faithful simulators. Given only a sequence of target oil painting images, can a robot infer and execute the stroke trajectories, forces, and colors needed to reproduce it? We present IMPASTO, a robotic oil-painting system that integrates learned pixel dynamics models with model-based planning. The dynamics models predict canvas updates from image observations and parameterized stroke actions; a receding-horizon model predictive control optimizer then plans trajectories and forces, while a force-sensitive controller executes strokes on a 7-DoF robot arm. IMPASTO integrates low-level force control, learned dynamics models, and high-level closed-loop planning, learns solely from robot self-play, and approximates human artists' single-stroke datasets and multi-stroke artworks, outperforming baselines in reproduction accuracy. Project website: https://impasto-robopainting.github.io/

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 31 Mar 2026.

Builds on

36 of 54 references.

References · 36

  1. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,656
  2. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,082
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,620
  4. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,334
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,241
  6. 2022DayDreamer: World Models for Physical Robot Learningcited by 375
  7. 2016Deep Visual Foresight for Planning Robot Motioncited by 281
  8. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 270
  9. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 259
  10. 2018Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Controlcited by 224
  11. 2018Learning Particle Dynamics for Manipulating Rigid Bodies, Deformable Objects, and Fluidscited by 219
  12. 2019Deep Dynamics Models for Learning Dexterous Manipulationcited by 201
24 more
  1. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 154
  2. 2017Self-Supervised Visual Planning with Temporal Skip Connectionscited by 139
  3. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  4. 2020Causal Discovery in Physical Systems from Videoscited by 77
  5. 2023RoboCook: Long-Horizon Elasto-Plastic Object Manipulation with Diverse Toolscited by 76
  6. 2022RoboCraft: Learning to See, Simulate, and Shape Elasto-Plastic Objects with Graph Networkscited by 70
  7. 2020Keypoints into the Future: Self-Supervised Correspondence in Model-Based Reinforcement Learningcited by 69
  8. 2018Propagation Networks for Model-Based Control Under Partial Observationcited by 64
  9. 2021Paint Transformer: Feed Forward Neural Painting with Stroke Predictioncited by 61
  10. 2024AdaptiGraph: Material-Adaptive Graph-Based Neural Dynamics for Robotic Manipulationcited by 53
  11. 2020The Surprising Effectiveness of Linear Models for Visual Foresight in Object Pile Manipulationcited by 25
  12. 2023Dynamic-Resolution Model Learning for Object Pile Manipulationcited by 22
  13. 2019Experience-Embedded Visual Foresightcited by 21
  14. 2022FRIDA: A Collaborative Robot Painter with a Differentiable, Real2Sim2Real Planning Environmentcited by 18
  15. 2020Content Masked Loss: Human-Like Brush Stroke Planning in a Reinforcement Learning Painting Agentcited by 15
  16. 2023Model Predictive Control for Aggressive Driving Over Uneven Terraincited by 8
  17. 2024CoFRIDA: Self-Supervised Fine-Tuning for Human-Robot Co-Paintingcited by 8
  18. 2023Deep Model Predictive Optimizationcited by 7
  19. 2022PaintNet: Unstructured Multi-Path Learning from 3D Point Clouds for Robotic Spray Paintingcited by 5
  20. 2024Spline-FRIDA: Towards Diverse, Humanlike Robot Painting Styles with a Sample-Efficient, Differentiable Brush Stroke Modelcited by 5
  21. 2024BaB-ND: Long-Horizon Motion Planning with Branch-and-Bound and Neural Dynamicscited by 5
  22. 2020Robot Calligraphy using Pseudospectral Optimal Control in Conjunction with a Novel Dynamic Brush Modelcited by 4
  23. 2022From Scratch to Sketch: Deep Decoupled Hierarchical Reinforcement Learning for Robotic Sketching Agentcited by 2
  24. 2023Sim-to-Real Brush Manipulation using Behavior Cloning and Reinforcement Learningcited by 1

Built on by

Not cited yet.