# Grounding Language Models in Autonomous Loco-manipulation Tasks

arXiv: https://arxiv.org/abs/2409.01326 · HTML: https://mlsanity.com/paper/2409.01326/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 3 references that are in the map.
- First posted on arXiv 2 Sep 2024.

## Builds on (most cited first, 3 of 3)

- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model](https://mlsanity.com/paper/2408.08282/) (2024)
- [HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation](https://mlsanity.com/paper/2406.14655/) (2024)

## Built on by (most cited first, 1 of 1)

- [INTENTION: Inferring Tendencies of Humanoid Robot Motion Through Interactive Intuition and Grounded VLM](https://mlsanity.com/paper/2508.04931/) (2025)

## Abstract (arXiv)

Humanoid robots with behavioral autonomy have consistently been regarded as ideal collaborators in our daily lives and promising representations of embodied intelligence. Compared to fixed-based robotic arms, humanoid robots offer a larger operational space while significantly increasing the difficulty of control and planning. Despite the rapid progress towards general-purpose humanoid robots, most studies remain focused on locomotion ability with few investigations into whole-body coordination and tasks planning, thus limiting the potential to demonstrate long-horizon tasks involving both mobility and manipulation under open-ended verbal instructions. In this work, we propose a novel framework that learns, selects, and plans behaviors based on tasks in different scenarios. We combine reinforcement learning (RL) with whole-body optimization to generate robot motions and store them into a motion library. We further leverage the planning and reasoning features of the large language model (LLM), constructing a hierarchical task graph that comprises a series of motion primitives to bridge lower-level execution with higher-level planning. Experiments in simulation and real-world using the CENTAURO robot show that the language model based planner can efficiently adapt to new loco-manipulation tasks, demonstrating high autonomy from free-text commands in unstructured scenes.