mlsanityData to 7 Oct 2026

Leveraging Language for Accelerated Learning of Tool Manipulation

arXiv 2206.13074 ↗First posted 27 Jun 2022cs.RO cs.AI cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Robust and generalized tool manipulation requires an understanding of the properties and affordances of different tools. We investigate whether linguistic information about a tool (e.g., its geometry, common uses) can help control policies adapt faster to new tools for a given task. We obtain diverse descriptions of various tools in natural language and use pre-trained language models to generate their feature representations. We then perform language-conditioned meta-learning to learn policies that can efficiently adapt to new tools given their corresponding text descriptions. Our results demonstrate that combining linguistic information and meta-learning significantly accelerates tool learning in several manipulation tasks including pushing, lifting, sweeping, and hammering.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 26 of its 47 references that are in the map. First posted on arXiv 27 Jun 2022.

Builds on

26 of 47 references.

References · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2016Layer Normalizationcited by 5,579
  3. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,861
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  6. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
  7. 2018On First-Order Meta-Learning Algorithmscited by 1,003
  8. 2019Meta-Learning with Implicit Gradientscited by 523
  9. 2018Meta-Learning for Semi-Supervised Few-Shot Classificationcited by 456
  10. 2018Probabilistic Model-Agnostic Meta-Learningcited by 254
  11. 2019Using Natural Language for Reward Shaping in Reinforcement Learningcited by 114
  12. 2019Multimodal Model-Agnostic Meta-Learning via Task-Aware Modulationcited by 103
14 more
  1. 2019TapNet: Neural Network Augmented with Task-Adaptive Projection for Few-Shot Learningcited by 90
  2. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80
  3. 2018Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervisioncited by 50
  4. 2021Grounding Language to Entities and Dynamics for Generalization in Reinforcement Learningcited by 45
  5. 2020Learning Rewards from Linguistic Feedbackcited by 41
  6. 2014Learning to Win by Reading Manuals in a Monte-Carlo Frameworkcited by 39
  7. 2018Guiding Policies with Language via Meta-Learningcited by 34
  8. 2019Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresightcited by 34
  9. 2019RTFM: Generalising to Novel Environment Dynamics via Readingcited by 33
  10. 2020Safe Reinforcement Learning with Natural Language Constraintscited by 22
  11. 2020Learning Adaptive Language Interfaces through Decompositioncited by 22
  12. 2019Meta Reinforcement Learning with Task Embedding and Shared Policycited by 15
  13. 2020Robust and Interpretable Grounding of Spatial References with Relation Networkscited by 8
  14. 2020Connecting Context-specific Adaptation in Humans to Meta-learningcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 10; 2025: 4; 2026: 2.

26 papers.

Sort:

By area:

  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  2. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  3. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  4. 2023Creative Robot Tool Use with Large Language Modelscited by 42
  5. 2024PoCo: Policy Composition from and for Heterogeneous Robot Learningcited by 42
  6. 2023Statler: State-Maintaining Language Models for Embodied Reasoningcited by 40
  7. 2023CoTDet: Affordance Knowledge Prompting for Task Driven Object Detectioncited by 36
  8. 2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learningcited by 36
  9. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  10. 2023Accelerating Reinforcement Learning of Robotic Manipulations via Feedback from Large Language Modelscited by 22
  11. 2023Transfer Knowledge from Natural Language to Electrocardiography: Can We Detect Cardiovascular Disease Through Language Models?cited by 21
  12. 2023AdaptSim: Task-Driven Simulation Adaptation for Sim-to-Real Transfercited by 17
14 more
  1. 2023Transfer Learning in Robotics: An Upcoming Breakthrough? A Review of Promises and Challengescited by 17
  2. 2024Language-Driven 6-DoF Grasp Detection Using Negative Prompt Guidancecited by 16
  3. 2024MANGO: A Benchmark for Evaluating Mapping and Navigation Abilities of Large Language Modelscited by 9
  4. 2024Retrieval-Augmented Hierarchical in-Context Reinforcement Learning and Hindsight Modular Reflections for Task Planning with LLMscited by 8
  5. 2024LINGO-Space: Language-Conditioned Incremental Grounding for Spacecited by 7
  6. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  7. 2025KUDA: Keypoints to Unify Dynamics Learning and Visual Prompting for Open-Vocabulary Robotic Manipulationcited by 5
  8. 2025CLASP: General-Purpose Clothes Manipulation with Semantic Keypointscited by 1
  9. 2023RL3: Boosting Meta Reinforcement Learning via RL inside RL2cited by 0
  10. 2024Emotion-Agent: Unsupervised Deep Reinforcement Learning with Distribution-Prototype Reward for Continuous Emotional EEG Analysiscited by 0
  11. 2025Watch Less, Feel More: Sim-to-Real RL for Generalizable Articulated Object Manipulation via Motion Adaptation and Impedance Controlcited by 0
  12. 2025Prolonging Tool Life: Learning Skillful Use of General-purpose Tools through Lifespan-guided Reinforcement Learningcited by 0
  13. 2026Creative Robot Tool Use by Counterfactual Reasoningcited by 0
  14. 2026GROW2: Grounding Which and Where for Robot Tool Usecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  2. 2026IROSA: Interactive Robot Skill Adaptation using Natural Language
  3. 2024Task Success Prediction for Open-Vocabulary Manipulation Based on Multi-Level Aligned Representations
  4. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  5. 2025CodeDiffuser: Attention-Enhanced Diffusion Policy via VLM-Generated Code for Instruction Ambiguity
  6. 2024CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision
  7. 2020Language-Conditioned Imitation Learning for Robot Manipulation Tasks
  8. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models