ABot-AgentOS: A General Robotic Agent OS with Lifelong Multi-modal Memory
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Recent VLM and VLA systems have improved robotic perception and action prediction, yet long-horizon embodied agents still require a general runtime layer for reasoning, memory, tool use, verification, and cross-embodiment execution. We present ABot-AgentOS, a general robotic Agent Operating System that sits above low-level controllers and provides a deliberative agent layer for scene-conditioned planning, context-isolated skill execution, multi-stage verification, multi-modal memory, and edge-cloud collaboration. To evaluate such systems, we introduce EmbodiedWorldBench, an executable benchmark with 16 indoor, outdoor, and hybrid scenes, four difficulty levels, and over 200 tasks involving navigation, object search, NPC dialogue, dynamic events, and trace-grounded scoring. ABot-AgentOS further introduces Universal Multi-modal Graph Memory, a persistent source-grounded substrate that converts dialogue, visual observations, spatial context, temporal relations, and task traces into typed nodes and edges. A failure-driven self-evolution loop converts diagnosed memory failures into gated runtime evo-assets that are promoted only to later evaluation splits, preventing current-split ground-truth leakage while enabling continual improvement. On an initial EmbodiedWorldBench subset, ABot-AgentOS improves over a single-controller baseline in both task success and goal completion. Across memory benchmarks, ABot-AgentOS Static achieves 87.5 on LoCoMo, 59.9 on OpenEQA EM-EQA, 88.6 on Mem-Gallery, and 76.5 Acc@All on NExT-QA; self-evolution further improves LoCoMo to 88.7, OpenEQA to 60.4, and Mem-Gallery to 89.0. These results suggest that a general Agent OS layer can improve long-horizon embodied execution while providing persistent, auditable memory for continual interaction.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 99 of its 114 references that are in the map. First posted on arXiv 11 Jul 2026.
Builds on
99 of 114 references.
Closest ancestors · 10
Same direction, newest first
- 2026When Robots Do the Chores: A Benchmark and Agent for Long-Horizon Household Task Execution14 May
- 2026Explore Like Humans: Autonomous Exploration with Online SG-Memo Construction for Embodied Agents21 Apr
- 2026Agentic Very Long Video Understanding26 Jan
- 2025WorldMM: Dynamic Multimodal Memory Agent for Long Video Reasoning2 Dec
- 2025RoboMemory: A Brain-inspired Multi-memory Agentic Framework for Interactive Environmental Learning in Physical Embodied Systems2 Aug
- 2025Embodied Web Agents: Bridging Physical-Digital Realms for Integrated Agent Intelligence18 Jun
- 2025Agentic Robot: A Brain-Inspired Framework for Vision-Language-Action Models in Embodied Agents29 May
- 2025EmbodiedBench: Comprehensive Benchmarking Multi-modal Large Language Models for Vision-Driven Embodied Agents13 Feb
- 2024EmbodiedCity: A Benchmark Platform for Embodied Agent in Real-world City Environment12 Oct
- 2024VideoAgent: A Memory-augmented Multimodal Agent for Video Understanding18 Mar
Other references · 89
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
- 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,988
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
- 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
38 more
- 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,081
- 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
- 2023Large Language Models as Optimizerscited by 864
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 832
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
- 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
- 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 744
- 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
- 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
- 2025Group-in-Group Policy Optimization for LLM Agent Trainingcited by 497
- 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
- 2020Just Ask: Learning to Answer Questions from Millions of Narrated Videoscited by 248
- 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
- 2025Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Modelscited by 221
- 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 220
- 2023A Simple LLM Framework for Long-Range Video Question-Answeringcited by 178
- 2026Qwen3.5-Omni Technical Reportcited by 174
- 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 173
- 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
- 2022Revisiting the "Video" in Video-Language Understandingcited by 169
- 2025WebAgent-R1: Training Web Agents via End-to-End Multi-Turn Reinforcement Learningcited by 165
- 2025Memory OS of AI Agentcited by 150
- 2026SimpleMem: Efficient Lifelong Memory for LLM Agentscited by 149
- 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 142
- 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 136
- 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
- 2026ABot-M0: VLA Foundation Model for Robotic Manipulation with Action Manifold Learningcited by 103
- 2023AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learncited by 97
- 2025MemEvolve: Meta-Evolution of Agent Memory Systemscited by 95
- 2025Galaxea Open-World Dataset and G0 Dual-System VLA Modelcited by 92
- 2025EgoLife: Towards Egocentric Life Assistantcited by 91
- 2025RoboBrain 2.0 Technical Reportcited by 89
- 2025Gemini Robotics 1.5: Pushing the Frontier of Generalist Robots with Advanced Embodied Reasoning, Thinking, and Motion Transfercited by 87
50 most cited of 89 All 89 (JSON)
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026ABot-Claw: A Foundation for Persistent, Cooperative, and Self-Evolving Robotic Agents
- 2026HoloAgent-0: A Unified Embodied Agent Framework with 3D Spatial Memory
- 2025RoboOS-NeXT: A Unified Memory-based Framework for Lifelong, Scalable, and Robust Multi-Robot Collaboration
- 2025FindingDory: A Benchmark to Evaluate Memory in Embodied Agents
- 2026PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution
- 2026What Should Embodied Agents Remember? Benchmarking and Evaluating Memory Modality for Embodied Agents
- 2026ChunkAct: Action Chunking for Agentic Robotics
- 2026RoboFoundry: System-as-Policy Evolution for Self-Learning Embodied Agents