mlsanityData to 14 Oct 2026

Affordance-Based Manipulation Planning with Text Goals and Sim-to-Real Generalisation via Real-to-Sim Image Conversion

arXiv 2607.11004 ↗First posted 13 Jul 2026cs.RO cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We present a manipulation planning system based on affordance recognition and action effect prediction. The system reasons through possible futures in visual form, and evaluates candidate plans by agreement of predicted outcomes with text-based goals set at run-time, using a multi-modal goal-matching module. Positions of objects named in the goal text are tracked through predictions even when occluded, making it possible to generate action plans even when objects become occluded, or when their initial descriptors cease to identify them in future states. We further expand the system with an image conversion module for translating real-world state images with objects of varied shapes and visual appearances into a consistent visual appearance, to facilitate manipulation planning in a physical robot setup. We evaluate performance of the system's modules in isolation and demonstrate the integrated system's manipulation planning capabilities on a set of challenging tasks in both simulation and on hardware.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 38 references that are in the map. First posted on arXiv 13 Jul 2026.

Builds on

23 of 38 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  7. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,222
  8. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  9. 2018signSGD: Compressed Optimisation for Non-Convex Problemscited by 571
  10. 2018Albumentations: fast and flexible image augmentationscited by 560
  11. 2018Unity: A General Platform for Intelligent Agentscited by 236
  12. 2020Scaled-YOLOv4: Scaling Cross Stage Partial Networkcited by 211
11 more
  1. 2023Vision-Language Models as Success Detectorscited by 142
  2. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  3. 2014Reducing the Barrier to Entry of Complex Robotic Software: a MoveIt! Case Studycited by 89
  4. 2016Learning Physical Intuition of Block Towers by Examplecited by 68
  5. 2020Deep Affordance Foresight: Planning Through What Can Be Done in the Futurecited by 41
  6. 2023Grasp-Anything: Large-scale Grasp Dataset from Foundation Modelscited by 33
  7. 2021Search-Based Task Planning with Learned Skill Effect Models for Lifelong Robotic Manipulationcited by 20
  8. 2022Generalizable Task Planning through Representation Pretrainingcited by 19
  9. 2023Multi-step planning with learned effects of partial action executionscited by 4
  10. 2023Multi-Object Graph Affordance Network: Goal-Oriented Planning through Learned Compound Object Affordancescited by 2
  11. 2022Recognising Affordances in Predicted Futures to Plan with Consideration of Non-canonical Affordance Effectscited by 1

Built on by

Not cited yet.