# Speech2Grasp: Data-Efficient Transfer of Text-Conditioned Grasp Detection to Speech in Humanoid Robots

arXiv: https://arxiv.org/abs/2607.26567 · HTML: https://mlsanity.com/paper/2607.26567/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 29 Jul 2026.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [LVIS: A Dataset for Large Vocabulary Instance Segmentation](https://mlsanity.com/paper/1908.03195/) (2019)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [LanguageBind: Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment](https://mlsanity.com/paper/2310.01852/) (2023)
- [OneLLM: One Framework to Align All Modalities with Language](https://mlsanity.com/paper/2312.03700/) (2023)
- [End-to-End Speech Recognition: A Survey](https://mlsanity.com/paper/2303.03329/) (2023)
- [Wavelet Diffusion Models are fast and scalable Image Generators](https://mlsanity.com/paper/2211.16152/) (2022)
- [Grasp-Anything: Large-scale Grasp Dataset from Foundation Models](https://mlsanity.com/paper/2309.09818/) (2023)
- [Language-driven Grasp Detection](https://mlsanity.com/paper/2406.09489/) (2024)
- [Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation](https://mlsanity.com/paper/2312.10807/) (2023)
- [Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidance](https://mlsanity.com/paper/2407.13842/) (2024)
- [SSR: Alignment-Aware Modality Connector for Speech Language Models](https://mlsanity.com/paper/2410.00168/) (2024)
- [GraspSAM: When Segment Anything Model Meets Grasp Detection](https://mlsanity.com/paper/2409.12521/) (2024)
- [LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation](https://mlsanity.com/paper/2502.20583/) (2025)
- [DWTNeRF: Boosting Few-shot Neural Radiance Fields via Discrete Wavelet Transform](https://mlsanity.com/paper/2501.12637/) (2025)
- [From Coarse to Fine: Learnable Discrete Wavelet Transforms for Efficient 3D Gaussian Splatting](https://mlsanity.com/paper/2506.23042/) (2025)
- [DWTGS: Rethinking Frequency Regularization for Sparse-view 3D Gaussian Splatting](https://mlsanity.com/paper/2507.15690/) (2025)
- [PoGaIN: Poisson-Gaussian Image Noise Modeling from Paired Samples](https://mlsanity.com/paper/2210.04866/) (2022)
- [Adaptive Knowledge Distillation between Text and Speech Pre-trained Models](https://mlsanity.com/paper/2303.03600/) (2023)
- [WaveletGaussian: Wavelet-domain Diffusion for Sparse-view 3D Gaussian Object Reconstruction](https://mlsanity.com/paper/2509.19073/) (2025)
- [AFD-SLU: Adaptive Feature Distillation for Spoken Language Understanding](https://mlsanity.com/paper/2509.04821/) (2025)
- [Learnable Multi-level Discrete Wavelet Transforms for 3D Gaussian Splatting Frequency Modulation](https://mlsanity.com/paper/2602.14199/) (2026)

## Abstract (arXiv)

Humanoid robots increasingly require multi-modal understanding for natural interaction with humans. Despite the prominence of vision-language models, they generally assume textual rather than the more natural speech inputs. In this paper, we investigate whether a well-established text-conditioned model can be transferred to speech in a data-efficient manner. Using ALBEF as a case study, we conduct diagnostic analyses showing that a lightweight MLP-based projector effectively adapts it to speech, while preserving semantic discrimination and robustness. Motivated by these findings, we introduce Speech2Grasp, a framework for data-efficient transfer of text-conditioned grasp detection to speech. Real-world humanoid robot experiments show that Speech2Grasp outperforms cascaded ASR-based pipeline, while reducing inference latency. Our findings suggest a practical paradigm for extending established text-conditioned systems to speech.