mlsanityData to 7 Oct 2026

Spatial-Language Attention Policies for Efficient Robot Learning

arXiv 2304.11235 ↗First posted 21 Apr 2023cs.RO cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Despite great strides in language-guided manipulation, existing work has been constrained to table-top settings. Table-tops allow for perfect and consistent camera angles, properties are that do not hold in mobile manipulation. Task plans that involve moving around the environment must be robust to egocentric views and changes in the plane and angle of grasp. A further challenge is ensuring this is all true while still being able to learn skills efficiently from limited data. We propose Spatial-Language Attention Policies (SLAP) as a solution. SLAP uses three-dimensional tokens as the input representation to train a single multi-task, language-conditioned action prediction policy. Our method shows an 80% success rate in the real world across eight tasks with a single model, and a 47.5% success rate when unseen clutter and unseen object configurations are introduced, even with only a handful of examples per task. This represents an improvement of 30% over prior work (20% given unseen distractors and configurations). We see a 4x improvement over baseline in mobile manipulation setting. In addition, we show how SLAPs robustness allows us to execute Task Plans from open-vocabulary instructions using a large language model for multi-step mobile manipulation. For videos, see the website: https://robotslap.github.io

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 45 of its 55 references that are in the map. First posted on arXiv 21 Apr 2023.

Builds on

45 of 55 references.

References · 45

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  7. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  12. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 671
33 more
  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  2. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  3. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 613
  4. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  5. 2023Larger language models do in-context learning differentlycited by 386
  6. 2022Visual Language Maps for Robot Navigationcited by 362
  7. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  8. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  9. 2017Dex-Net 2.0: Deep Learning to Plan Robust Grasps with Synthetic Point Clouds and Analytic Grasp Metricscited by 259
  10. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  11. 2021Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenescited by 213
  12. 2021Where2Act: From Pixels to Actions for Articulated 3D Objectscited by 179
  13. 2022CLIP-Fields: Weakly Supervised Semantic Fields for Robotic Memorycited by 164
  14. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  15. 2021FILM: Following Instructions in Language with Modular Methodscited by 119
  16. 2022Grounding Language with Visual Affordances over Unstructured Datacited by 101
  17. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  18. 2021A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Executioncited by 91
  19. 2020Unseen Object Instance Segmentation for Robotic Environmentscited by 85
  20. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  21. 2021VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objectscited by 78
  22. 20196-DOF Grasping for Target-driven Object Manipulation in Cluttercited by 70
  23. 2020Learning RGB-D Feature Embeddings for Unseen Object Instance Segmentationcited by 64
  24. 2021UMPNet: Universal Manipulation Policy Network for Articulated Objectscited by 64
  25. 2021StructFormer: Learning Spatial Structure for Language-Guided Semantic Rearrangement of Novel Objectscited by 57
  26. 2021SORNet: Spatial Object-Centric Representations for Sequential Manipulationcited by 48
  27. 2022IFOR: Iterative Flow Minimization for Robotic Object Rearrangementcited by 42
  28. 2021The Design of Stretch: A Compact, Lightweight Mobile Manipulator for Indoor Human Environmentscited by 39
  29. 2021Long-Horizon Manipulation of Unknown Objects via Task and Motion Planning with Estimated Affordancescited by 38
  30. 2021NeRP: Neural Rearrangement Planning for Unknown Objectscited by 33
  31. 2021Predicting Stable Configurations for Semantic Placement of Novel Objectscited by 23
  32. 2019Prospection: Interpretable Plans From Language By Predicting the Futurecited by 21
  33. 2023The Framework Tax: Disparities Between Inference Efficiency in NLP Research and Deploymentcited by 11

Built on by

Cited by year

Citing papers per year: 2024: 3.

3 papers.

  1. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  2. 2024Learning Generalizable Feature Fields for Mobile Manipulationcited by 31
  3. 2024Object Segmentation from Open-Vocabulary Manipulation Instructions Based on Optimal Transport Polygon Matching with Multimodal Foundation Modelscited by 1