# Language-Conditioned Semantic Search-Based Policy for Robotic Manipulation Tasks

arXiv: https://arxiv.org/abs/2312.05925 · HTML: https://mlsanity.com/paper/2312.05925/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Language-conditioned policies and VLAs, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 14 of its 19 references that are in the map.
- First posted on arXiv 10 Dec 2023.

## Placed in

- Offline RL and imitation learning from data / [Language-conditioned policies and VLAs](https://mlsanity.com/offlinerl/language-conditioned-policies-and-vlas/)

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Grounded Language-Image Pre-training](https://mlsanity.com/paper/2112.03857/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [Fast Segment Anything](https://mlsanity.com/paper/2306.12156/) (2023)
- [Language Conditioned Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2005.07648/) (2020)
- [What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Data](https://mlsanity.com/paper/2204.06252/) (2022)
- [Contrastive Language, Action, and State Pre-training for Robot Learning](https://mlsanity.com/paper/2304.10782/) (2023)
- [Critic Guided Segmentation of Rewarding Objects in First-Person Views](https://mlsanity.com/paper/2107.09540/) (2021)
- [Behavioral Cloning via Search in Video PreTraining Latent Space](https://mlsanity.com/paper/2212.13326/) (2022)
- [Planning with RL and episodic-memory behavioral priors](https://mlsanity.com/paper/2207.01845/) (2022)
- [Solving Learn-to-Race Autonomous Racing Challenge by Planning in Latent Space](https://mlsanity.com/paper/2207.01275/) (2022)
- [Shape complexity estimation using VAE](https://mlsanity.com/paper/2304.02766/) (2023)
- [Behavioral Cloning via Search in Embedded Demonstration Dataset](https://mlsanity.com/paper/2306.09082/) (2023)

## Built on by (most cited first, 5 of 5)

- [RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation](https://mlsanity.com/paper/2407.04689/) (2024)
- [STRAP: Robot Sub-Trajectory Retrieval for Augmented Policy Learning](https://mlsanity.com/paper/2412.15182/) (2024)
- [UniTeam: Open Vocabulary Mobile Manipulation Challenge](https://mlsanity.com/paper/2312.08611/) (2023)
- [Benchmarks for Physical Reasoning AI](https://mlsanity.com/paper/2312.10728/) (2023)
- [SplatR : Experience Goal Visual Rearrangement with 3D Gaussian Splatting and Dense Feature Matching](https://mlsanity.com/paper/2411.14322/) (2024)

## Abstract (arXiv)

Reinforcement learning and Imitation Learning approaches utilize policy learning strategies that are difficult to generalize well with just a few examples of a task. In this work, we propose a language-conditioned semantic search-based method to produce an online search-based policy from the available demonstration dataset of state-action trajectories. Here we directly acquire actions from the most similar manipulation trajectories found in the dataset. Our approach surpasses the performance of the baselines on the CALVIN benchmark and exhibits strong zero-shot adaptation capabilities. This holds great potential for expanding the use of our online search-based policy approach to tasks typically addressed by Imitation Learning or Reinforcement Learning-based policies.