mlsanityData to 14 Oct 2026

KALIE: Fine-Tuning Vision-Language Models for Open-World Manipulation without Robot Data

arXiv 2409.14066 ↗First posted 21 Sep 2024cs.RO cs.AI cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

Abstract (arXiv)

Building generalist robotic systems involves effectively endowing robots with the capabilities to handle novel objects in an open-world setting. Inspired by the advances of large pre-trained models, we propose Keypoint Affordance Learning from Imagined Environments (KALIE), which adapts pre-trained Vision Language Models (VLMs) for robotic control in a scalable manner. Instead of directly producing motor commands, KALIE controls the robot by predicting point-based affordance representations based on natural language instructions and visual observations of the scene. The VLM is trained on 2D images with affordances labeled by humans, bypassing the need for training data collected on robotic systems. Through an affordance-aware data synthesis pipeline, KALIE automatically creates massive high-quality training data based on limited example data manually collected by humans. We demonstrate that KALIE can learn to robustly solve new manipulation tasks with unseen objects given only 50 example data points. Compared to baselines using pre-trained VLMs, our approach consistently achieves superior performance.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 47 references that are in the map. First posted on arXiv 21 Sep 2024.

Builds on

39 of 47 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learning16 May
  2. 2024Vision-Language Models Provide Promptable Representations for Reinforcement Learning5 Feb
  3. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control28 Jul
  5. 2023PaLM-E: An Embodied Multimodal Language Model6 Mar

Other references · 34

  1. 2023Segment Anythingcited by 8,234
  2. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  4. 2024Octo: An Open-Source Generalist Robot Policycited by 1,484
  5. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,210
  6. 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
  7. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  8. 2019Solving Rubik's Cube with a Robot Handcited by 734
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  10. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  11. 2020Reinforcement Learning with Augmented Datacited by 468
  12. 2023Learning Interactive Real-World Simulatorscited by 458
22 more
  1. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 439
  2. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  3. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  4. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 299
  5. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  6. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
  7. 2015Supersizing Self-supervision: Learning to Grasp from 50K Tries and 700 Robot Hourscited by 209
  8. 2021Fake It Till You Make It: Face analysis in the wild using synthetic data alonecited by 200
  9. 2023Scaling Robot Learning with Semantically Imagined Experiencecited by 177
  10. 2019Active Domain Randomizationcited by 170
  11. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  12. 2019kPAM: KeyPoint Affordances for Category-Level Robotic Manipulationcited by 153
  13. 2024MOKA: Open-World Robotic Manipulation through Mark-Based Visual Promptingcited by 129
  14. 2019Meta-Sim: Learning to Generate Synthetic Datasetscited by 128
  15. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 118
  16. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Roboticscited by 106
  17. 2023GenAug: Retargeting behaviors to unseen situations via Generative Augmentationcited by 104
  18. 2022BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotationscited by 99
  19. 2023General In-Hand Object Rotation with Vision and Touchcited by 99
  20. 2023NeuralField-LDM: Scene Generation with Hierarchical Latent Diffusion Modelscited by 76
  21. 2020Meta-Sim2: Unsupervised Learning of Scene Structure for Synthetic Data Generationcited by 58
  22. 2023Tiny and Efficient Model for the Edge Detection Generalizationcited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 8; 2026: 5.

15 papers.

Sort:

By area:

  1. 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challengescited by 47
  2. 2025ManipBench: Benchmarking Vision-Language Models for Low-Level Robot Manipulationcited by 24
  3. 2024Blox-Net: Generative Design-for-Robot-Assembly Using VLM Supervision, Physics Simulation, and a Robot with Resetcited by 15
  4. 2025Spatial RoboGrasp: Generalized Robotic Grasping Control Policycited by 6
  5. 2025A Real-to-Sim-to-Real Approach to Robotic Manipulation with VLM-Generated Iterative Keypoint Rewardscited by 5
  6. 2025D-CODA: Diffusion for Coordinated Dual-Arm Data Augmentationcited by 5
  7. 2025RoboGrasp: A Universal Grasping Policy for Robust Robotic Controlcited by 4
  8. 2025HybridGen: VLM-Guided Hybrid Planning for Scalable Data Generation of Imitation Learningcited by 3
  9. 2024AugInsert: Learning Robust Visual-Force Policies via Data Augmentation for Object Assembly Taskscited by 2
  10. 2026RoboVista: Evaluating Vision Language Models for Diverse Robot Applicationscited by 2
  11. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?cited by 1
  12. 2026Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learningcited by 1
3 more
  1. 2025BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generationcited by 0
  2. 2026On the Strengths and Weaknesses of Data for Open-set Embodied Assistancecited by 0
  3. 2026What Objects Enable, Not What They Are: Functional Latent Spaces for Affordance Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Learning Instruction-Guided Manipulation Affordance via Large Models for Embodied Robotic Tasks
  2. 2026AffordanceVLA: A Vision-Language-Action Model Empowering Action Generation through Affordance-Aware Understanding
  3. 2025AffordGrasp: In-Context Affordance Reasoning for Open-Vocabulary Task-Oriented Grasping in Clutter
  4. 2025Affordance Field Intervention: Enabling VLAs to Escape Memory Traps in Robotic Manipulation
  5. 2024Affordance-Guided Reinforcement Learning via Visual Prompting
  6. 2025UAD: Unsupervised Affordance Distillation for Generalization in Robotic Manipulation
  7. 2026AffordSim: A Scalable Data Generator and Benchmark for Affordance-Aware Robotic Manipulation
  8. 2024CoA-VLA: Improving Vision-Language-Action Models via Visual-Textual Chain-of-Affordance