# Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping

arXiv: https://arxiv.org/abs/2406.15677 · HTML: https://mlsanity.com/paper/2406.15677/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Jun 2024.

## Built on by (most cited first, 10 of 10)

- [Equivariant Action Sampling for Reinforcement Learning and Planning](https://mlsanity.com/paper/2412.12237/) (2024)
- [IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks](https://mlsanity.com/paper/2509.24768/) (2025)
- [Lifelong Language-Conditioned Robotic Manipulation Learning](https://mlsanity.com/paper/2603.05160/) (2026)
- [Efficient Alignment of Unconditioned Action Prior for Language-conditioned Pick and Place in Clutter](https://mlsanity.com/paper/2503.09423/) (2025)
- [Optimal Interactive Learning on the Job via Facility Location Planning](https://mlsanity.com/paper/2505.00490/) (2025)
- [Eq.Bot: Enhance Robotic Manipulation Learning via Group Equivariant Canonicalization](https://mlsanity.com/paper/2511.15194/) (2025)
- [Robust Instruction Compliance in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2605.12655/) (2026)
- [Bounding Boxes as Goals: Language-Conditioned Grasping via Neuro-Symbolic Planning](https://mlsanity.com/paper/2606.12910/) (2026)
- [Action Map Policy: Learning 3D Closed-loop Manipulation via Pixel Classification](https://mlsanity.com/paper/2607.10706/) (2026)
- [Pix2Act: Image-Space Manipulation Policies with Equivariant Augmentation](https://mlsanity.com/paper/2607.11167/) (2026)

## Abstract (arXiv)

Controlling robots through natural language is pivotal for enhancing human-robot collaboration and synthesizing complex robot behaviors. Recent works that are trained on large robot datasets show impressive generalization abilities. However, such pretrained methods are (1) often fragile to unseen scenarios, and (2) expensive to adapt to new tasks. This paper introduces Grounded Equivariant Manipulation (GEM), a robust yet efficient approach that leverages pretrained vision-language models with equivariant language mapping for language-conditioned manipulation tasks. Our experiments demonstrate GEM's high sample efficiency and generalization ability across diverse tasks in both simulation and the real world. GEM achieves similar or higher performance with orders of magnitude fewer robot data compared with major data-efficient baselines such as CLIPort and VIMA. Finally, our approach demonstrates greater robustness compared to large VLA model, e.g, OpenVLA, at correctly interpreting natural language commands on unseen objects and poses. Code, data, and training details are available https://saulbatman.github.io/gem_page/