# Improving Generalization of Language-Conditioned Robot Manipulation

arXiv: https://arxiv.org/abs/2508.02405 · HTML: https://mlsanity.com/paper/2508.02405/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 29 references that are in the map.
- First posted on arXiv 4 Aug 2025.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [ConceptFusion: Open-set Multimodal 3D Mapping](https://mlsanity.com/paper/2302.07241/) (2023)
- [Convolutions Die Hard: Open-Vocabulary Segmentation with Single Frozen Convolutional CLIP](https://mlsanity.com/paper/2308.02487/) (2023)
- [MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting](https://mlsanity.com/paper/2403.03174/) (2024)
- [UP-VLA: A Unified Understanding and Prediction Model for Embodied Agent](https://mlsanity.com/paper/2501.18867/) (2025)
- [CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models](https://mlsanity.com/paper/2403.08248/) (2024)
- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [IGOR: Image-GOal Representations are the Atomic Control Units for Foundation Models in Embodied AI](https://mlsanity.com/paper/2411.00785/) (2024)
- [LP++: A Surprisingly Strong Linear Probe for Few-Shot CLIP](https://mlsanity.com/paper/2404.02285/) (2024)
- [Programmatically Grounded, Compositionally Generalizable Robotic Manipulation](https://mlsanity.com/paper/2304.13826/) (2023)
- [Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models](https://mlsanity.com/paper/2312.04533/) (2023)
- [Task-Oriented Grasp Prediction with Visual-Language Inputs](https://mlsanity.com/paper/2302.14355/) (2023)
- [Vision-Language Model Fine-Tuning via Simple Parameter-Efficient Modification](https://mlsanity.com/paper/2409.16718/) (2024)
- [LM4LV: A Frozen Large Language Model for Low-level Vision Tasks](https://mlsanity.com/paper/2405.15734/) (2024)
- [Proto-CLIP: Vision-Language Prototypical Network for Few-Shot Learning](https://mlsanity.com/paper/2307.03073/) (2023)
- [Visual Grounding with Multi-modal Conditional Adaptation](https://mlsanity.com/paper/2409.04999/) (2024)

## Built on by (most cited first, 1 of 1)

- [Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning](https://mlsanity.com/paper/2606.12910/) (2026)

## Abstract (arXiv)

The control of robots for manipulation tasks generally relies on visual input. Recent advances in vision-language models (VLMs) enable the use of natural language instructions to condition visual input and control robots in a wider range of environments. However, existing methods require a large amount of data to fine-tune VLMs for operating in unseen environments. In this paper, we present a framework that learns object-arrangement tasks from just a few demonstrations. We propose a two-stage framework that divides object-arrangement tasks into a target localization stage, for picking the object, and a region determination stage for placing the object. We present an instance-level semantic fusion module that aligns the instance-level image crops with the text embedding, enabling the model to identify the target objects defined by the natural language instructions. We validate our method on both simulation and real-world robotic environments. Our method, fine-tuned with a few demonstrations, improves generalization capability and demonstrates zero-shot ability in real-robot manipulation scenarios.