# Long-horizon Locomotion and Manipulation on a Quadrupedal Robot with Large Language Models

arXiv: https://arxiv.org/abs/2404.05291 · HTML: https://mlsanity.com/paper/2404.05291/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Apr 2024.

## Built on by (most cited first, 21 of 21)

- [NaVILA: Legged Robot Vision-Language-Action Model for Navigation](https://mlsanity.com/paper/2412.04453/) (2024)
- [One Policy to Run Them All: an End-to-end Learning Approach to Multi-Embodiment Locomotion](https://mlsanity.com/paper/2409.06366/) (2024)
- [Autonomous Improvement of Instruction Following Skills via Foundation Models](https://mlsanity.com/paper/2407.20635/) (2024)
- [CurricuLLM: Automatic Task Curricula Design for Learning Complex Robot Skills using Large Language Models](https://mlsanity.com/paper/2409.18382/) (2024)
- [Commonsense Reasoning for Legged Robot Adaptation with Vision-Language Models](https://mlsanity.com/paper/2407.02666/) (2024)
- [QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning](https://mlsanity.com/paper/2412.15576/) (2024)
- [Safe LLM-Controlled Robots with Formal Guarantees via Reachability Analysis](https://mlsanity.com/paper/2503.03911/) (2025)
- [QuaDreamer: Controllable Panoramic Video Generation for Quadruped Robots](https://mlsanity.com/paper/2508.02512/) (2025)
- [Deep Reinforcement Learning for Bipedal Locomotion: A Brief Survey](https://mlsanity.com/paper/2404.17070/) (2024)
- [QuadrupedGPT: Towards a Versatile Quadruped Agent in Open-ended Worlds](https://mlsanity.com/paper/2406.16578/) (2024)
- [From Reward Shaping to Q-Shaping: Achieving Unbiased Learning with LLM-Guided Knowledge](https://mlsanity.com/paper/2410.01458/) (2024)
- [Bipedalism for Quadrupedal Robots: Versatile Loco-Manipulation through Risk-Adaptive Reinforcement Learning](https://mlsanity.com/paper/2507.20382/) (2025)
- [Drift is a Sampling Error: SNR-Aware Power Distributions for Long-Horizon Robotic Planning](https://mlsanity.com/paper/2605.09537/) (2026)
- [Enhancing Q-Learning with Large Language Model Heuristics](https://mlsanity.com/paper/2405.03341/) (2024)
- [AINav: Large Language Model-Based Adaptive Interactive Navigation](https://mlsanity.com/paper/2503.22942/) (2025)
- [Deep Dive into Model-free Reinforcement Learning for Biological and Robotic Systems: Theory and Practice](https://mlsanity.com/paper/2405.11457/) (2024)
- [PEPA: a Persistently Autonomous Embodied Agent with Personalities](https://mlsanity.com/paper/2603.00117/) (2026)
- [OpenGo: An OpenClaw-Based Robotic Dog with Real-Time Skill Switching](https://mlsanity.com/paper/2604.01708/) (2026)
- [Off-Policy Actor-Critic with Sigmoid-Bounded Entropy for Real-World Robot Learning](https://mlsanity.com/paper/2601.15761/) (2026)
- [StepTrigger: Contact-State-Triggered Backdoor Attacks on VLM-Powered Legged Robots](https://mlsanity.com/paper/2609.26131/) (2026)
- [Silent Sabotage: Internal State Triggered Backdoor Attacks on LLM-Powered Robotic Systems](https://mlsanity.com/paper/2609.26184/) (2026)

## Abstract (arXiv)

We present a large language model (LLM) based system to empower quadrupedal robots with problem-solving abilities for long-horizon tasks beyond short-term motions. Long-horizon tasks for quadrupeds are challenging since they require both a high-level understanding of the semantics of the problem for task planning and a broad range of locomotion and manipulation skills to interact with the environment. Our system builds a high-level reasoning layer with large language models, which generates hybrid discrete-continuous plans as robot code from task descriptions. It comprises multiple LLM agents: a semantic planner that sketches a plan, a parameter calculator that predicts arguments in the plan, a code generator that converts the plan into executable robot code, and a replanner that handles execution failures or human interventions. At the low level, we adopt reinforcement learning to train a set of motion planning and control skills to unleash the flexibility of quadrupeds for rich environment interactions. Our system is tested on long-horizon tasks that are infeasible to complete with one single skill. Simulation and real-world experiments show that it successfully figures out multi-step strategies and demonstrates non-trivial behaviors, including building tools or notifying a human for help. Demos are available on our project page: https://sites.google.com/view/long-horizon-robot.