mlsanityData to 7 Oct 2026

LLM as A Robotic Brain: Unifying Egocentric Memory and Control

arXiv 2304.09349 ↗First posted 19 Apr 2023cs.AI cs.CL cs.ROcited by 32

Not on the map. Why?

Abstract (arXiv)

Embodied AI focuses on the study and development of intelligent systems that possess a physical or virtual embodiment (i.e. robots) and are able to dynamically interact with their environment. Memory and control are the two essential parts of an embodied system and usually require separate frameworks to model each of them. In this paper, we propose a novel and generalizable framework called LLM-Brain: using Large-scale Language Model as a robotic brain to unify egocentric memory and control. The LLM-Brain framework integrates multiple multimodal language models for robotic tasks, utilizing a zero-shot learning approach. All components within LLM-Brain communicate using natural language in closed-loop multi-round dialogues that encompass perception, planning, control, and memory. The core of the system is an embodied LLM to maintain egocentric memory and control the robot. We demonstrate LLM-Brain by examining two downstream tasks: active exploration and embodied question answering. The active exploration tasks require the robot to extensively explore an unknown environment within a limited number of actions. Meanwhile, the embodied question answering tasks necessitate that the robot answers questions based on observations acquired during prior explorations.

Key facts. Cited by 32 papers in the map (data through 7 Oct 2026). Builds on 23 of its 25 references that are in the map. First posted on arXiv 19 Apr 2023.

Builds on

23 of 25 references.

References · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  3. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  7. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  8. 2019Habitat: A Platform for Embodied AI Researchcited by 1,163
  9. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  11. 2022A Generalist Agentcited by 802
  12. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
11 more
  1. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Languagecited by 513
  2. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  3. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  4. 2023Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agentscited by 396
  5. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  6. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  7. 2022Instruction-driven history-aware policies for robotic manipulationscited by 93
  8. 2023ChatGPT Asks, BLIP-2 Answers: Automatic Questioning Towards Enriched Visual Descriptionscited by 86
  9. 2022See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulationcited by 61
  10. 2023ChatGPT Empowered Long-Step Robot Control in Various Environments: A Case Applicationcited by 48
  11. 2023Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptionscited by 42

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 9; 2025: 10; 2026: 3.

32 papers.

Sort:

By area:

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  2. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  3. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  4. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  5. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  6. 2024SplitLoRA: A Split Parameter-Efficient Fine-Tuning Framework for Large Language Modelscited by 60
  7. 2024BadRobot: Jailbreaking Embodied LLM Agents in the Physical Worldcited by 40
  8. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Modelscited by 40
  9. 2025Visual Large Language Models for Generalized and Specialized Applicationscited by 35
  10. 2023Towards Knowledge-driven Autonomous Drivingcited by 29
  11. 2025World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planningcited by 26
  12. 2023Holistic Evaluation of GPT-4V for Biomedical Imagingcited by 19
20 more
  1. 2024LLM-Personalize: Aligning LLM Planners with Human Preferences via Reinforced Self-Training for Housekeeping Robotscited by 18
  2. 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluationcited by 17
  3. 2023Why Can Large Language Models Generate Correct Chain-of-Thoughts?cited by 16
  4. 2024KoMA: Knowledge-driven Multi-agent Framework for Autonomous Driving with Large Language Modelscited by 16
  5. 2024Semantic Trajectory Data Mining with LLM-Informed POI Classificationcited by 13
  6. 2025World-aware Planning Narratives Enhance Large Vision-Language Model Plannercited by 9
  7. 2025The Mirage of Performance Gains: Why Contrastive Decoding Fails to Mitigate Object Hallucinations in MLLMs?cited by 7
  8. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  9. 2023WALL-E: Embodied Robotic WAiter Load Lifting with Large Language Modelcited by 6
  10. 2024Polaris: Open-ended Interactive Robotic Manipulation via Syn2Real Visual Grounding and Large Language Modelscited by 6
  11. 2024Stephanie: Step-by-Step Dialogues for Mimicking Human Interactions in Social Conversationscited by 5
  12. 2023SwitchGPT: Adapting Large Language Models for Non-Text Outputscited by 4
  13. 2024LLMs for Robotic Object Disambiguationcited by 4
  14. 2025Benchmarking LLM Privacy Recognition for Social Robot Decision Makingcited by 3
  15. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priorscited by 3
  16. 2023The Role of Summarization in Generative Agents: A Preliminary Perspectivecited by 2
  17. 2025Tuning LLMs by RAG Principles: Towards LLM-native Memorycited by 2
  18. 2025Chat with UAV -- Human-UAV Interaction Based on Large Language Modelscited by 1
  19. 2026MemCtrl: Using MLLMs as Active Memory Controllers on Embodied Agentscited by 1
  20. 2026Human-Inspired Context-Selective Multimodal Memory for Social Robotscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Language to Action: Can LLM-Based Agents Be Used for Embodied Robot Cognition?
  2. 2026Athena-Brain Technical Report: An Efficient Robot Brain for General Intelligence and Embodied Interaction
  3. 2024Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation
  4. 2024Embodied-RAG: General Non-parametric Embodied Memory for Retrieval and Generation
  5. 2025Embodied AI: From LLMs to World Models
  6. 2023Open-Ended Instructable Embodied Agents with Memory-Augmented Large Language Models
  7. 2025Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence
  8. 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systems