mlsanityData to 7 Oct 2026

A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations): A Visual-Symbolic Framework for Virtual Humans

arXiv 2609.35463 ↗First posted 28 Sep 2026cs.AI cs.GRcited by 0

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

Creating believable vh requires the coherent integration of perception, reasoning, and action mediated by language. A central challenge is to combine these components into a control loop grounded in interactive 3D environments. To this end, we present A.D.A.M.O. (Agent for language-Driven Actions with Multimodal Observations), a visual-symbolic framework for language-driven vh that leverages a pretrained vlm with tool calling to unify perception, reasoning, and action within a single control loop. A.D.A.M.O. maintains a dual visual-symbolic world model that combines egocentric visual input and synchronized symbolic state to support grounded task-oriented behavior from natural language prompts. To support diagnostic evaluation, we introduce a controlled task suite organized by a cd taxonomy that breaks down spatial tasks into procedural and linguistic complexity. Experiments in controlled scenes show that semantic labeling strongly influences task completion and failure modes, reducing perceptual ambiguity while shifting failures toward downstream execution, whereas reasoning errors remain comparatively rare.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 12 of its 32 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

12 of 32 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024KARMA: Augmenting Embodied AI Agents with Long-and-short Term Memory Systems23 Sep

Other references · 11

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  5. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 773
  6. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  7. 2021Open-Ended Learning Leads to Generally Capable Agentscited by 177
  8. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  9. 2023JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Modelscited by 170
  10. 2023See and Think: Embodied Agent in Virtual Environmentcited by 54
  11. 2024Collaborative Quest Completion with LLM-driven Non-Player Characters in Minecraftcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents
  2. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models
  3. 2025Probing a Vision-Language-Action Model for Symbolic States and Integration into a Cognitive Architecture
  4. 2026Towards Neuro-Symbolic Procedural Reasoning for Long-Horizon Vision-Language-Action Manipulation
  5. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspective
  6. 2025Igniting VLMs toward the Embodied Space
  7. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective
  8. 2024A Survey on Vision-Language-Action Models for Embodied AI