# Agent as Cerebrum, Controller as Cerebellum: Implementing an Embodied LMM-based Agent on Drones

arXiv: https://arxiv.org/abs/2311.15033 · HTML: https://mlsanity.com/paper/2311.15033/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 40 references that are in the map.
- First posted on arXiv 25 Nov 2023.

## Builds on (most cited first, 25 of 25)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning](https://mlsanity.com/paper/1910.10897/) (2019)
- [HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face](https://mlsanity.com/paper/2303.17580/) (2023)
- [Open-vocabulary Object Detection via Vision and Language Knowledge Distillation](https://mlsanity.com/paper/2104.13921/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation](https://mlsanity.com/paper/1806.10293/) (2018)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Learning Latent Plans from Play](https://mlsanity.com/paper/1903.01973/) (2019)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Deep Imitation Learning for Complex Manipulation Tasks from Virtual Reality Teleoperation](https://mlsanity.com/paper/1710.04615/) (2017)
- [MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale](https://mlsanity.com/paper/2104.08212/) (2021)
- [DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Reality](https://mlsanity.com/paper/2210.13702/) (2022)
- [Vision-Language Pre-training: Basics, Recent Advances, and Future Trends](https://mlsanity.com/paper/2210.09263/) (2022)
- [V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control](https://mlsanity.com/paper/1909.12238/) (2019)
- [CACTI: A Framework for Scalable Multi-Task Multi-Scene Visual Imitation Learning](https://mlsanity.com/paper/2212.05711/) (2022)
- [Large Language Model-Aware In-Context Learning for Code Generation](https://mlsanity.com/paper/2310.09748/) (2023)
- [Learning to Push by Grasping: Using multiple tasks for effective learning](https://mlsanity.com/paper/1609.09025/) (2016)
- [Improving Data Efficiency of Self-supervised Learning for Robotic Grasping](https://mlsanity.com/paper/1903.00228/) (2019)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)

## Built on by (most cited first, 9 of 9)

- [UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobility](https://mlsanity.com/paper/2501.02341/) (2025)
- [Towards Efficient Online Tuning of VLM Agents via Counterfactual Soft Reinforcement Learning](https://mlsanity.com/paper/2505.03792/) (2025)
- [CHAI: Command Hijacking against embodied AI](https://mlsanity.com/paper/2510.00181/) (2025)
- [Leveraging YOLO-World and GPT-4V LMMs for Zero-Shot Person Detection and Action Recognition in Drone Imagery](https://mlsanity.com/paper/2404.01571/) (2024)
- [Embodied Crowd Counting](https://mlsanity.com/paper/2503.08367/) (2025)
- [If you're waiting for a sign... that might not be it! Mitigating Trust Boundary Confusion from Visual Injections on Vision-Language Agentic Systems](https://mlsanity.com/paper/2604.19844/) (2026)
- [LEO-RobotAgent: A General-purpose Robotic Agent for Language-driven Embodied Operator](https://mlsanity.com/paper/2512.10605/) (2025)
- [Beyond Attack Success Rate: Examining Trigger Leakage in Vision-Language Agentic Systems](https://mlsanity.com/paper/2606.12586/) (2026)
- [Vision Language Action (VLA) Models for Unmanned Aerial Robotics and Bimanual Manipulation: A Review](https://mlsanity.com/paper/2607.06706/) (2026)

## Abstract (arXiv)

In this study, we present a novel paradigm for industrial robotic embodied agents, encapsulating an 'agent as cerebrum, controller as cerebellum' architecture. Our approach harnesses the power of Large Multimodal Models (LMMs) within an agent framework known as AeroAgent, tailored for drone technology in industrial settings. To facilitate seamless integration with robotic systems, we introduce ROSchain, a bespoke linkage framework connecting LMM-based agents to the Robot Operating System (ROS). We report findings from extensive empirical research, including simulated experiments on the Airgen and real-world case study, particularly in individual search and rescue operations. The results demonstrate AeroAgent's superior performance in comparison to existing Deep Reinforcement Learning (DRL)-based agents, highlighting the advantages of the embodied LMM in complex, real-world scenarios.