mlsanityData to 7 Oct 2026

AVA: Towards Autonomous Visualization Agents through Visual Perception-Driven Decision-Making

arXiv 2312.04494 ↗First posted 7 Dec 2023cs.HC cs.AI cs.CVcited by 21

On the map: LLM agents and tool use → Multimodal and embodied agents

Abstract (arXiv)

With recent advances in multi-modal foundation models, the previously text-only large language models (LLM) have evolved to incorporate visual input, opening up unprecedented opportunities for various applications in visualization. Our work explores the utilization of the visual perception ability of multi-modal LLMs to develop Autonomous Visualization Agents (AVAs) that can interpret and accomplish user-defined visualization objectives through natural language. We propose the first framework for the design of AVAs and present several usage scenarios intended to demonstrate the general applicability of the proposed paradigm. The addition of visual perception allows AVAs to act as the virtual visualization assistant for domain experts who may lack the knowledge or expertise in fine-tuning visualization outputs. Our preliminary exploration and proof-of-concept agents suggest that this approach can be widely applicable whenever the choices of appropriate visualization parameters require the interpretation of previous visual output. Feedback from unstructured interviews with experts in AI research, medical visualization, and radiology has been incorporated, highlighting the practicality and potential of AVAs. Our study indicates that AVAs represent a general paradigm for designing intelligent visualization systems that can achieve high-level visualization goals, which pave the way for developing expert-level visualization agents in the future.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 25 of its 43 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

25 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models8 Dec

Other references · 24

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  7. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  8. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  9. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  12. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 591
12 more
  1. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 337
  2. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 300
  3. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 231
  4. 2023LIDA: A Tool for Automatic Generation of Grammar-Agnostic Visualizations and Infographics using Large Language Modelscited by 80
  5. 2020NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queriescited by 73
  6. 2018Data2Vis: Automatic Generation of Data Visualizations Using Sequence to Sequence Recurrent Neural Networkscited by 47
  7. 2023LLM4Vis: Explainable Visualization Recommendation using ChatGPTcited by 25
  8. 2021KG4Vis: A Knowledge Graph-Based Approach for Visualization Recommendationcited by 17
  9. 2023Beyond Generating Code: Evaluating GPT on a Data Visualization Coursecited by 16
  10. 2022Rich Screen Reader Experiences for Accessible Data Visualizationcited by 12
  11. 2022Facilitating Conversational Interaction in Natural Language Interfaces for Visualizationcited by 12
  12. 2022Adapting Zeroth Order Algorithms for Comparison-Based Optimizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 10; 2026: 9.

21 papers.

Sort:

By area:

  1. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 58
  2. 2024WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualizationcited by 38
  3. 2025NLI4VolVis: Natural Language Interaction for Volume Visualization via LLM Multi-Agents and Editable 3D Gaussian Splattingcited by 19
  4. 2025ProactiveVA: Proactive Visual Analytics with LLM-Based UI Agentcited by 14
  5. 2025Agentic Visualization: Extracting Agent-based Design Patterns from Visualization Systemscited by 12
  6. 2025ParaView-MCP: An Autonomous Visualization Agent with Direct Tool Usecited by 11
  7. 2024Visualization Literacy of Multimodal Large Language Models: A Comparative Studycited by 9
  8. 2026SciVisAgentBench: A Benchmark for Evaluating Scientific Data Analysis and Visualization Agentscited by 9
  9. 2025See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMscited by 7
  10. 2025An Evaluation-Centric Paradigm for Scientific Visualization Agentscited by 6
  11. 2025IntuiTF: MLLM-Guided Transfer Function Optimization for Direct Volume Renderingcited by 4
  12. 2026TopoPilot: Reliable Conversational Workflow Automation for Topological Data Analysis and Visualizationcited by 3
9 more
  1. 2026Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualizationcited by 2
  2. 2026Toward AI VIS Co-Scientists: A General and End-to-End Agent Harness for Solving Complex Data Visualization Taskscited by 2
  3. 2025A Generative AI System for Biomedical Data Discovery with Grammar-Based Visualizationscited by 1
  4. 2025YAC: Bridging Natural Language and Interactive Visual Exploration with Generative AI for Biomedical Data Discoverycited by 1
  5. 2026Animating Petascale Time-varying Data on Commodity Hardware with LLM-assisted Scriptingcited by 1
  6. 2026SciVisAgentSkills: Design and Evaluation of Agent Skills for Scientific Data Analysis and Visualizationcited by 1
  7. 2026HiLSVA: Design and Evaluation of a Human-in-the-Loop Agentic System for Scientific Visualizationcited by 1
  8. 2026Context-Mediated Domain Adaptation in Multi-Agent Sensemaking Systemscited by 0
  9. 2026MedVA: An End-to-End Neuro-Symbolic Agentic System for Medical Volume Visualizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Do Vision-Language Models See Visualizations Like Humans? Alignment in Chart Categorization
  2. 2024The Visualization JUDGE : Can Multimodal Foundation Models Guide Visualization Design Through Visual Perception?
  3. 2025Orion: A Unified Visual Agent for Multimodal Perception, Advanced Visual Reasoning and Execution
  4. 2025VizGenie: Toward Self-Refining, Domain-Aware Workflows for Next-Generation Scientific Visualization
  5. 2026MultiVis-Agent: A Multi-Agent Framework with Logic Rules for Reliable and Comprehensive Cross-Modal Data Visualization
  6. 2025A Composable Agentic System for Automated Visual Data Reporting
  7. 2026Do MLLMs See What We See? Analyzing Visualization Literacy Barriers in AI Systems
  8. 2025Do Language Model Agents Align with Humans in Rating Visualizations? An Empirical Study