# Humanoid Agent via Embodied Chain-of-Action Reasoning with Multimodal Foundation Models for Zero-Shot Loco-Manipulation

arXiv: https://arxiv.org/abs/2504.09532 · HTML: https://mlsanity.com/paper/2504.09532/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 13 Apr 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 20 of 20)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [RMA: Rapid Motor Adaptation for Legged Robots](https://mlsanity.com/paper/2107.04034/) (2021)
- [Modular Multitask Reinforcement Learning with Policy Sketches](https://mlsanity.com/paper/1611.01796/) (2016)
- [HumanoidBench: Simulated Humanoid Benchmark for Whole-Body Locomotion and Manipulation](https://mlsanity.com/paper/2403.10506/) (2024)
- [Skill Induction and Planning with Latent Language](https://mlsanity.com/paper/2110.01517/) (2021)
- [PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interaction](https://mlsanity.com/paper/2312.04393/) (2023)
- [Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real World](https://mlsanity.com/paper/2110.05457/) (2021)
- [Hierarchical Planning and Control for Box Loco-Manipulation](https://mlsanity.com/paper/2306.09532/) (2023)
- [Fully Autonomous Real-World Reinforcement Learning with Applications to Mobile Manipulation](https://mlsanity.com/paper/2107.13545/) (2021)
- [Grasp-Anything: Large-scale Grasp Dataset from Foundation Models](https://mlsanity.com/paper/2309.09818/) (2023)
- [GAMap: Zero-Shot Object Goal Navigation with Multi-Scale Geometric-Affordance Guidance](https://mlsanity.com/paper/2410.23978/) (2024)
- [Zero-shot Object Navigation with Vision-Language Models Reasoning](https://mlsanity.com/paper/2410.18570/) (2024)
- [RS-MoE: A Vision-Language Model with Mixture of Experts for Remote Sensing Image Captioning and Visual Question Answering](https://mlsanity.com/paper/2411.01595/) (2024)
- [Autonomous Behavior Planning For Humanoid Loco-manipulation Through Grounded Language Model](https://mlsanity.com/paper/2408.08282/) (2024)
- [Socially-Aware Robot Navigation Enhanced by Bidirectional Natural Language Conversations Using Large Language Models](https://mlsanity.com/paper/2409.04965/) (2024)

## Built on by (most cited first, 3 of 3)

- [TRACE: Textual Reasoning for Affordance Coordinate Extraction](https://mlsanity.com/paper/2511.01999/) (2025)
- [From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Models](https://mlsanity.com/paper/2511.12861/) (2025)
- [InterEvolve: Test-Time Evolution of Reward Programs for Humanoid Loco-Manipulation](https://mlsanity.com/paper/2610.02196/) (2026)

## Abstract (arXiv)

Humanoid loco-manipulation, which integrates whole-body locomotion with dexterous manipulation, remains a fundamental challenge in robotics. Beyond whole-body coordination and balance, a central difficulty lies in understanding human instructions and translating them into coherent sequences of embodied actions. Recent advances in foundation models provide transferable multimodal representations and reasoning capabilities, yet existing efforts remain largely restricted to either locomotion or manipulation in isolation, with limited applicability to humanoid settings. In this paper, we propose Humanoid-COA, the first humanoid agent framework that integrates foundation model reasoning with an Embodied Chain-of-Action (CoA) mechanism for zero-shot loco-manipulation. Within the perception--reasoning--action paradigm, our key contribution lies in the reasoning stage, where the proposed CoA mechanism decomposes high-level human instructions into structured sequences of locomotion and manipulation primitives through affordance analysis, spatial inference, and whole-body action reasoning. Extensive experiments on two humanoid robots, Unitree H1-2 and G1, in both an open test area and an apartment environment, demonstrate that our framework substantially outperforms prior baselines across manipulation, locomotion, and loco-manipulation tasks, achieving robust generalization to long-horizon and unstructured scenarios. Project page: https://humanoid-coa.github.io/