mlsanityData to 7 Oct 2026

AnySkill: Learning Open-Vocabulary Physical Skill for Interactive Agents

arXiv 2403.12835 ↗First posted 19 Mar 2024cs.CV cs.ROcited by 32

On the map: Offline RL and imitation learning from data → Robot manipulation imitation learning

Abstract (arXiv)

Traditional approaches in physics-based motion generation, centered around imitation learning and reward shaping, often struggle to adapt to new scenarios. To tackle this limitation, we propose AnySkill, a novel hierarchical method that learns physically plausible interactions following open-vocabulary instructions. Our approach begins by developing a set of atomic actions via a low-level controller trained via imitation learning. Upon receiving an open-vocabulary textual instruction, AnySkill employs a high-level policy that selects and integrates these atomic actions to maximize the CLIP similarity between the agent's rendered images and the text. An important feature of our method is the use of image-based rewards for the high-level policy, which allows the agent to learn interactions with objects without manual reward engineering. We demonstrate AnySkill's capability to generate realistic and natural motion sequences in response to unseen instructions of varying lengths, marking it the first method capable of open-vocabulary physical skill learning for interactive humanoid agents.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Placed in Robot manipulation imitation learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 77 references that are in the map. First posted on arXiv 19 Mar 2024.

Builds on

53 of 77 references.

References · 53

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  5. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  6. 2022Human Motion Diffusion Modelcited by 1,062
  7. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  8. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  9. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  10. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  11. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  12. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
38 more
  1. 2022EDGE: Editable Dance Generation From Musiccited by 284
  2. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  3. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenescited by 237
  4. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  5. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  6. 2021Stochastic Scene-Aware Motion Predictioncited by 203
  7. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  8. 2023Vision-Language Models are Zero-Shot Reward Models for Reinforcement Learningcited by 155
  9. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  10. 2020Synthesizing Long-Term 3D Human Motion and Interaction in 3D Scenescited by 135
  11. 2019Generating 3D People in Scenes without Peoplecited by 127
  12. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  13. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  14. 2022Recovering 3D Human Mesh from Monocular Images: A Surveycited by 103
  15. 2023Synthesizing Physical Character-Scene Interactionscited by 102
  16. 20233D Human Pose Estimation via Intuitive Physicscited by 100
  17. 2018Multi-Agent Generative Adversarial Imitation Learningcited by 99
  18. 2018Neural probabilistic motor primitives for humanoid controlcited by 96
  19. 2023CALM: Conditional Adversarial Latent Models for Directable Virtual Characterscited by 91
  20. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 82
  21. 2022Full-Body Articulated Human-Object Interactioncited by 76
  22. 2018Human-centric Indoor Scene Synthesis Using Stochastic Grammarcited by 73
  23. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  24. 2021Scene-aware Generative Network for Human Motion Synthesiscited by 70
  25. 2021Physics-based Human Motion Estimation and Synthesis from Videoscited by 65
  26. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  27. 2022COAP: Compositional Articulated Occupancy of Peoplecited by 59
  28. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  29. 2022MIME: Human-Aware 3D Scene Generationcited by 57
  30. 2022Towards Understanding Mixture of Experts in Deep Learningcited by 52
  31. 2023Locomotion-Action-Manipulation: Synthesizing Human-Scene Interactions in Complex 3D Environmentscited by 51
  32. 2023PADL: Language-Directed Physics-Based Character Controlcited by 47
  33. 2022Being Comes from Not-being: Open-vocabulary Text-to-Motion Generation with Wordless Trainingcited by 45
  34. 2022Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Modelcited by 41
  35. 2022Differentiable Dynamics for Articulated 3d Human Motion Reconstructioncited by 38
  36. 2023Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Modelscited by 37
  37. 2022Motron: Multimodal Probabilistic Human Motion Forecastingcited by 30
  38. 2023Composite Motion Learning with Task Controlcited by 26

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 10; 2025: 14; 2026: 8.

32 papers.

Sort:

By area:

  1. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  2. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  3. 2024InterDreamer: Zero-Shot Text to 3D Dynamic Human-Object Interactioncited by 63
  4. 2024Autonomous Character-Scene Interaction Synthesis from Text Instructioncited by 56
  5. 2024HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environmentcited by 52
  6. 2025TokenHSI: Unified Synthesis of Physical Human-Scene Interactions through Task Tokenizationcited by 46
  7. 2024PhyRecon: Physically Plausible Neural Scene Reconstructioncited by 44
  8. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  9. 2024SkillMimic: Learning Basketball Interaction Skills from Demonstrationscited by 25
  10. 2025ChainHOI: Joint-based Kinematic Chain Modeling for Human-Object Interaction Generationcited by 19
  11. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  12. 2024SCENIC: Scene-aware Semantic Navigation with Instruction-guided Controlcited by 13
20 more
  1. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  2. 2024MOVIS: Enhancing Multi-Object Novel View Synthesis for Indoor Scenescited by 9
  3. 2025GROVE: A Generalized Reward for Learning Open-Vocabulary Physical Skillcited by 9
  4. 2025AnyTask: an Automated Task and Data Generation Framework for Advancing Sim-to-Real Policy Learningcited by 8
  5. 2025ModSkill: Physical Character Skill Modularizationcited by 7
  6. 2025StickMotion: Generating 3D Human Motions by Drawing a Stickmancited by 7
  7. 2025UniAct: Unified Motion Generation and Action Streaming for Humanoid Robotscited by 5
  8. 2025PhysiInter: Integrating Physical Mapping for High-Fidelity Human Interaction Generationcited by 4
  9. 2025InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphscited by 4
  10. 2025Spatial-Temporal Multi-Scale Quantization for Flexible Motion Generationcited by 3
  11. 2024COLLAGE: Collaborative Human-Agent Interaction Generation using Hierarchical Latent Diffusion and Language Modelscited by 2
  12. 2025ForceGrip: Reference-Free Curriculum Learning for Realistic Grip Force Control in VR Hand Manipulationcited by 1
  13. 2026DeVI: Physics-based Dexterous Human-Object Interaction via Synthetic Video Imitationcited by 1
  14. 2026GIRAF: Towards Generalizable Human Interactions with Articulated Objectscited by 1
  15. 2025LHM-Humanoid: Long-Horizon Human Motion Control for Continuous Object Transport in Cluttered Scenescited by 0
  16. 2026MVR: Multi-view Video Reward Shaping for Reinforcement Learningcited by 0
  17. 2026Re2MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinementcited by 0
  18. 2026DrawMotion: Generating 3D Human Motions by Freehand Drawingcited by 0
  19. 2026DiscoForcing: A Unified Framework for Real-Time Audio-Driven Character Control with Diffusion Forcingcited by 0
  20. 2026In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026InterReal: A Unified Physics-Based Imitation Framework for Learning Human-Object Interaction Skills
  2. 2025NIL: No-data Imitation Learning by Leveraging Pre-trained Video Diffusion Models
  3. 2026MimicAgent: Quadruped Skills via Prompt-to-Trajectory Generation
  4. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play
  5. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Models
  6. 2025Learning Semantic Atomic Skills for Multi-Task Robotic Manipulation
  7. 2026HumanX: Toward Agile and Generalizable Humanoid Interaction Skills from Human Videos
  8. 2026Dual-Process Atomic Skill Learning: Decoupling Semantic Reasoning and Real-Time Control