# Semantic-Geometric-Physical-Driven Robot Manipulation Skill Transfer via Skill Library and Tactile Representation

arXiv: https://arxiv.org/abs/2411.11714 · HTML: https://mlsanity.com/paper/2411.11714/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 18 Nov 2024.

## Builds on (most cited first, 22 of 22)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Progressive Neural Networks](https://mlsanity.com/paper/1606.04671/) (2016)
- [ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulation](https://mlsanity.com/paper/2409.01652/) (2024)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [A Review of Robot Learning for Manipulation: Challenges, Representations, and Algorithms](https://mlsanity.com/paper/1907.03146/) (2019)
- [FurnitureBench: Reproducible Real-World Benchmark for Long-Horizon Complex Manipulation](https://mlsanity.com/paper/2305.12821/) (2023)
- [Behavior Trees in Robotics and AI: An Introduction](https://mlsanity.com/paper/1709.00084/) (2017)
- [Self-Supervised Learning for Domain Adaptation on Point-Clouds](https://mlsanity.com/paper/2003.12641/) (2020)
- [CoPa: General Robotic Manipulation through Spatial Constraints of Parts with Foundation Models](https://mlsanity.com/paper/2403.08248/) (2024)
- [Manipulate-Anything: Automating Real-World Robots using Vision-Language Models](https://mlsanity.com/paper/2406.18915/) (2024)
- [Learning Symmetric and Low-energy Locomotion](https://mlsanity.com/paper/1801.08093/) (2018)
- [Cable Manipulation with a Tactile-Reactive Gripper](https://mlsanity.com/paper/1910.02860/) (2019)
- [GraspGPT: Leveraging Semantic Knowledge from a Large Language Model for Task-Oriented Grasping](https://mlsanity.com/paper/2307.13204/) (2023)
- [GraspGF: Learning Score-based Grasping Primitive for Human-assisting Dexterous Grasping](https://mlsanity.com/paper/2309.06038/) (2023)
- [Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs](https://mlsanity.com/paper/2311.02847/) (2023)
- [Extrinsic Contact Sensing with Relative-Motion Tracking from Distributed Tactile Measurements](https://mlsanity.com/paper/2103.08108/) (2021)
- [Canonical Representation and Force-Based Pretraining of 3D Tactile for Dexterous Visuo-Tactile Policy Learning](https://mlsanity.com/paper/2409.17549/) (2024)
- [RoboCSE: Robot Common Sense Embedding](https://mlsanity.com/paper/1903.00412/) (2019)
- [Cable Routing and Assembly using Tactile-driven Motion Primitives](https://mlsanity.com/paper/2303.11765/) (2023)

## Built on by (most cited first, 1 of 1)

- [Cross-Document Contextual Coreference Resolution in Knowledge Graphs](https://mlsanity.com/paper/2504.05767/) (2025)

## Abstract (arXiv)

Developing general robotic systems capable of manipulating in unstructured environments is a significant challenge, particularly as the tasks involved are typically long-horizon and rich-contact, requiring efficient skill transfer across different task scenarios. To address these challenges, we propose knowledge graph-based skill library construction method. This method hierarchically organizes manipulation knowledge using "task graph" and "scene graph" to represent task-specific and scene-specific information, respectively. Additionally, we introduce "state graph" to facilitate the interaction between high-level task planning and low-level scene information. Building upon this foundation, we further propose a novel hierarchical skill transfer framework based on the skill library and tactile representation, which integrates high-level reasoning for skill transfer and low-level precision for execution. At the task level, we utilize large language models (LLMs) and combine contextual learning with a four-stage chain-of-thought prompting paradigm to achieve subtask sequence transfer. At the motion level, we develop an adaptive trajectory transfer method based on the skill library and the heuristic path planning algorithm. At the physical level, we propose an adaptive contour extraction and posture perception method based on tactile representation. This method dynamically acquires high-precision contour and posture information from visual-tactile images, adjusting parameters such as contact position and posture to ensure the effectiveness of transferred skills in new environments. Experiments demonstrate the skill transfer and adaptability capabilities of the proposed methods across different task scenarios. Project website: https://github.com/MingchaoQi/skill_transfer