Robo-Cortex: A Self-Evolving Embodied Agent via Dual-Grain Cognitive Memory and Autonomous Knowledge Induction
On the map: LLM agents and tool use → Multimodal and embodied agents
also in 1 more
Abstract (arXiv)
The ability to navigate and interact with complex environments is central to real-world embodied agents, yet navigation in unseen environments remains challenging due to "experiential amnesia," where existing trajectory-driven or reactive policies fail to synthesize generalizable strategies from past interactions. We propose Robo-Cortex, a self-evolving framework that enables robots to autonomously induce navigation heuristics and refine cognitive strategies through a continuous reflection-adaptation loop. By abstracting success patterns and failure pitfalls into natural-language heuristics, Robo-Cortex enables a transition from passive execution to active strategy evolution. Our core innovation is an Autonomous Knowledge Induction (AKI) mechanism that distills multimodal trajectories into a structured Navigation Heuristic Library for knowledge generalization. The architecture further incorporates a Dual-Grain Cognitive Memory system, comprising a Short-term Reflective Memory (SRM) for real-time local progress analysis, and a Long-term Principle Memory (LPM) that abstracts past trajectories into reusable guiding and cautionary principles. To ensure robust decision-making, we introduce a multimodal Imagine-then-Verify loop, where a world model simulates potential outcomes and a VLM-based evaluator validates action plans. Extensive evaluations on IGNav, AR, and AEQA show that Robo-Cortex consistently outperforms strong baselines in both task success and exploration efficiency, with gains of up to +4.16% SPL over the strongest prior method and up to +15.30% SPL under heuristic transfer to unseen environments. Preliminary real-world robotic experiments further support the effectiveness of Robo-Cortex in physical settings.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 44 of its 49 references that are in the map. First posted on arXiv 18 May 2026.
Builds on
44 of 49 references.
References · 44
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
- 2025Wan: Open and Advanced Large-Scale Video Generative Modelscited by 2,708
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
- 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
- 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
- 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
- 2021Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AIcited by 592
- 2020Object Goal Navigation using Goal-Oriented Semantic Explorationcited by 429
32 more
- 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
- 2017Embodied Question Answeringcited by 336
- 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
- 2023NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Modelscited by 280
- 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 271
- 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
- 2020Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-trainingcited by 231
- 2022Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigationcited by 193
- 2023Towards Learning a Generalist Model for Embodied Navigationcited by 138
- 2025EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecyclecited by 137
- 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
- 2024EvoAgent: Towards Automatic Multi-Agent Generation via Evolutionary Algorithmscited by 113
- 2023CLIN: A Continually Learning Language Agent for Rapid Task Adaptation and Generalizationcited by 95
- 2024MapGPT: Map-Guided Prompting with Adaptive Path Planning for Vision-and-Language Navigationcited by 86
- 20243D-Mem: 3D Scene Memory for Embodied Exploration and Reasoningcited by 75
- 2025Embodied Navigation Foundation Modelcited by 72
- 2024HM3D-OVON: A Dataset and Benchmark for Open-Vocabulary Object Goal Navigationcited by 71
- 2025MapNav: A Novel Memory Representation via Annotated Semantic Maps for Vision-and-Language Navigationcited by 60
- 2017A Dataset for Developing and Benchmarking Active Visioncited by 50
- 2019Bayesian Relational Memory for Semantic Visual Navigationcited by 48
- 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
- 2025WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigationcited by 43
- 2025Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigationcited by 30
- 2023Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Modelscited by 18
- 2024Devil's Advocate: Anticipatory Reflection for LLM Agentscited by 17
- 2024MemoNav: Working Memory Model for Visual Navigationcited by 13
- 2020Take the Scenic Route: Improving Generalization in Vision-and-Language Navigationcited by 10
- 2023ESceme: Vision-and-Language Navigation with Episodic Scene Memorycited by 10
- 2025C-NAV: Towards Self-Evolving Continual Object Navigation in Open Worldcited by 9
- 2024Planning from Imagination: Episodic Simulation and Episodic Memory for Vision-and-Language Navigationcited by 8
- 2024Vision-Language Navigation with Continual Learningcited by 6
- 2026Lifelong Embodied Navigation Learningcited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systems
- 2026BrainMem: Brain-Inspired Evolving Memory for Embodied Agent Task Planning
- 2025From reactive to cognitive: brain-inspired spatial intelligence for embodied agents
- 2026Reflect After Failure: Brain-Inspired Self-Evolving Agents for Embodied Spatial Intelligence
- 2026RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents
- 2026EmbodiedRSI: Active Continual Robot Learning Through Hypothesis-Guided Co-Evolution
- 2025Embodied World Models Emerge from Navigational Task in Open-Ended Environments
- 2026Explore Like Humans: Autonomous Exploration with Online SG-Memo Construction for Embodied Agents