# DaDu-E: Rethinking the Role of Large Language Model in Robotic Computing Pipeline

arXiv: https://arxiv.org/abs/2412.01663 · HTML: https://mlsanity.com/paper/2412.01663/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 29 of its 31 references that are in the map.
- First posted on arXiv 2 Dec 2024.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 29 of 29)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Code as Policies: Language Model Programs for Embodied Control](https://mlsanity.com/paper/2209.07753/) (2022)
- [On Evaluation of Embodied Navigation Agents](https://mlsanity.com/paper/1807.06757/) (2018)
- [Vision-Language Foundation Models as Effective Robot Imitators](https://mlsanity.com/paper/2311.01378/) (2023)
- [Large Language Models Are Reasoning Teachers](https://mlsanity.com/paper/2212.10071/) (2022)
- [AnyGrasp: Robust and Efficient Grasp Perception in Spatial and Temporal Domains](https://mlsanity.com/paper/2212.08333/) (2022)
- [Large Language Models are Versatile Decomposers: Decompose Evidence and Questions for Table-based Reasoning](https://mlsanity.com/paper/2301.13808/) (2023)
- [SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Models](https://mlsanity.com/paper/2309.10062/) (2023)
- [TaskBench: Benchmarking Large Language Models for Task Automation](https://mlsanity.com/paper/2311.18760/) (2023)
- [Task and Motion Planning with Large Language Models for Object Rearrangement](https://mlsanity.com/paper/2303.06247/) (2023)
- [Closed-Loop Visuomotor Control with Generative Expectation for Robotic Manipulation](https://mlsanity.com/paper/2409.09016/) (2024)
- [Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks](https://mlsanity.com/paper/2405.01534/) (2024)
- [RePLan: Robotic Replanning with Perception and Language Models](https://mlsanity.com/paper/2401.04157/) (2024)
- [CoPAL: Corrective Planning of Robot Actions with Large Language Models](https://mlsanity.com/paper/2310.07263/) (2023)
- [Closed-Loop Open-Vocabulary Mobile Manipulation with GPT-4V](https://mlsanity.com/paper/2404.10220/) (2024)
- [KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems](https://mlsanity.com/paper/2409.14908/) (2024)
- [DaDu-Corki: Algorithm-Architecture Co-Design for Embodied AI-powered Robotic Manipulation](https://mlsanity.com/paper/2407.04292/) (2024)
- [Closed-Loop Long-Horizon Robotic Planning via Equilibrium Sequence Modeling](https://mlsanity.com/paper/2410.01440/) (2024)

## Built on by (most cited first, 5 of 5)

- [How Fast Can I Run My VLA? Demystifying VLA Inference Performance with VLA-Perf](https://mlsanity.com/paper/2602.18397/) (2026)
- [SafeMind: Benchmarking and Mitigating Safety Risks in Embodied LLM Agents](https://mlsanity.com/paper/2509.25885/) (2025)
- [ANNIE: Be Careful of Your Robots](https://mlsanity.com/paper/2509.03383/) (2025)
- [KAITIAN: A Unified Communication Framework for Enabling Efficient Collaboration Across Heterogeneous Accelerators in Embodied AI Systems](https://mlsanity.com/paper/2505.10183/) (2025)
- [MALLVI: A Multi-Agent Framework for Integrated Generalized Robotics Manipulation](https://mlsanity.com/paper/2602.16898/) (2026)

## Abstract (arXiv)

Performing complex tasks in open environments remains challenging for robots, even when using large language models (LLMs) as the core planner. Many LLM-based planners are inefficient due to their large number of parameters and prone to inaccuracies because they operate in open-loop systems. We think the reason is that only applying LLMs as planners is insufficient. In this work, we propose DaDu-E, a robust closed-loop planning framework for embodied AI robots. Specifically, DaDu-E is equipped with a relatively lightweight LLM, a set of encapsulated robot skill instructions, a robust feedback system, and memory augmentation. Together, these components enable DaDu-E to (i) actively perceive and adapt to dynamic environments, (ii) optimize computational costs while maintaining high performance, and (iii) recover from execution failures using its memory and feedback mechanisms. Extensive experiments on real-world and simulated tasks show that DaDu-E achieves task success rates comparable to embodied AI robots with larger models as planners like COME-Robot, while reducing computational requirements by 6.6 ×. Users are encouraged to explore our system at: urlhttps://rlc-lab.github.io/dadu-e/.