mlsanityData to 7 Oct 2026

Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions

arXiv 2410.12773 ↗First posted 16 Oct 2024cs.RO cs.AIcited by 22

Not on the map. Why?

Abstract (arXiv)

Humanoid robots, with their human-like embodiment, have the potential to integrate seamlessly into human environments. Critical to their coexistence and cooperation with humans is the ability to understand natural language communications and exhibit human-like behaviors. This work focuses on generating diverse whole-body motions for humanoid robots from language descriptions. We leverage human motion priors from extensive human motion datasets to initialize humanoid motions and employ the commonsense reasoning capabilities of Vision Language Models (VLMs) to edit and refine these motions. Our approach demonstrates the capability to produce natural, expressive, and text-aligned humanoid motions, validated through both simulated and real-world experiments. More videos can be found at https://ut-austin-rpl.github.io/Harmon/.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 32 of its 57 references that are in the map. First posted on arXiv 16 Oct 2024.

Builds on

32 of 57 references.

References · 32

  1. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  2. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  3. 2022Human Motion Diffusion Modelcited by 1,062
  4. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  5. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  6. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  7. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  8. 2017On human motion prediction using recurrent neural networkscited by 336
  9. 2019Learning Trajectory Dependencies for Human Motion Predictioncited by 245
  10. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  11. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  12. 2020DLow: Diversifying Latent Flows for Diverse Human Motion Predictioncited by 184
20 more
  1. 2017HP-GAN: Probabilistic 3D human motion prediction via GANcited by 176
  2. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  3. 2023Generalized Planning in PDDL Domains with Pretrained Large Language Modelscited by 139
  4. 2024Expressive Whole-Body Control for Humanoid Robotscited by 117
  5. 2024Learning Human-to-Humanoid Real-Time Whole-Body Teleoperationcited by 112
  6. 2021SimPoE: Simulated Character Control for 3D Human Pose Estimationcited by 111
  7. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  8. 2018QuaterNet: A Quaternion-based Recurrent Model for Human Motioncited by 108
  9. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  10. 2019Structured Prediction Helps 3D Human Motion Modellingcited by 96
  11. 2018A Neural Temporal Model for Human Motion Predictioncited by 87
  12. 2019Ego-Pose Estimation and Forecasting as Real-Time PD Controlcited by 87
  13. 2018MT-VAE: Learning Motion Transformations to Generate Multimodal Human Dynamicscited by 86
  14. 2022QuestSim: Human Motion Tracking from Sparse Sensors with Simulated Avatarscited by 74
  15. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 39
  16. 2023Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thoughtcited by 39
  17. 2023Interactive Robot Learning from Verbal Correctioncited by 28
  18. 2022PoseTriplet: Co-evolving 3D Human Pose Estimation, Imitation, and Hallucination under Self-supervisioncited by 25
  19. 2023From Text to Motion: Grounding GPT-4 in a Humanoid Robot "Alter3"cited by 13
  20. 2023Words into Action: Learning Diverse Humanoid Robot Behaviors using Language Guided Iterative Motion Refinementcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 12; 2026: 8.

22 papers.

Sort:

By area:

  1. 2025Humanoid-VLA: Towards Universal Humanoid Control with Visual Integrationcited by 43
  2. 2025Learning Humanoid Standing-up Control across Diverse Posturescited by 34
  3. 2025BeamDojo: Learning Agile Humanoid Locomotion on Sparse Footholdscited by 32
  4. 2025BEHAVIOR Robot Suite: Streamlining Real-World Whole-Body Manipulation for Everyday Household Activitiescited by 29
  5. 2024Learning from Massive Human Videos for Universal Humanoid Pose Controlcited by 27
  6. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  7. 2025UniMedVL: Unifying Medical Multimodal Understanding and Generation through Observation-Knowledge-Analysiscited by 12
  8. 2026FRoM-W1: Towards General Humanoid Whole-Body Control with Language Instructionscited by 10
  9. 2025Perceiving and Acting in First-Person: A Dataset and Benchmark for Egocentric Human-Object-Human Interactionscited by 7
  10. 2025MoMaGen: Generating Demonstrations under Soft and Hard Constraints for Multi-Step Bimanual Mobile Manipulationcited by 7
  11. 2025Free-T2M: Robust Text-to-Motion Generation for Humanoid Robots via Frequency-Domaincited by 5
  12. 2025DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotioncited by 5
10 more
  1. 2025Natural Humanoid Robot Locomotion with Generative Motion Priorcited by 4
  2. 2025FlowMotion: Target-Predictive Conditional Flow Matching for Jitter-Reduced Text-Driven Human Motion Generationcited by 3
  3. 2026Towards Motion Turing Test: Evaluating Human-Likeness in Humanoid Robotscited by 3
  4. 2026PhysDrift: Bridging the Embodiment Gap in Humanoid Co-Speech Motion Generationcited by 3
  5. 2025Embodied Learning of Reward for Musculoskeletal Control with Vision Language Modelscited by 0
  6. 2026RobotDesignGPT: Automated Robot Design Synthesis using Vision Language Modelscited by 0
  7. 2026A Self-Supervised Approach on Motion Calibration for Enhancing Physical Plausibility in Text-to-Motioncited by 0
  8. 2026Re2MoGen: Open-Vocabulary Motion Generation via LLM Reasoning and Physics-Aware Refinementcited by 0
  9. 2026Before the Body Moves: Learning Anticipatory Joint Intent for Language-Conditioned Humanoid Controlcited by 0
  10. 2026From Sign Language Generation to Humanoid Execution: Vision-Language Guided Retargeting with Collision Mitigationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Commanding Humanoid by Free-form Language: A Large Language Action Model with Unified Motion Vocabulary
  2. 2025SENTINEL: A Fully End-to-End Language-Action Model for Humanoid Whole Body Control
  3. 2025Semantic Co-Speech Gesture Synthesis and Real-Time Control for Humanoid Robots ↗
  4. 2026EmoPose: Vision-Language Model Guided Emotion-Aware Gesture Generation for Humanoid Robots ↗
  5. 2026ECHO-G: Embodied Co-speech Humanoid mOtion Generation
  6. 2026WOLF-VLA: Whole-Body Humanoid Optimal Locomotion Framework for Vision-Language-Action Learning
  7. 2026RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction
  8. 2026WholeBodyWAM: Learning Whole-Body World Action Models with Scalable Motion Priors