Lexara-RF: Reference-Free Metrics for Evaluating Conversational Visual Analytics Agents
On the map: LLM agents and tool use → Multimodal and embodied agents
Abstract (arXiv)
Conversational visual analytics (CVA) agents powered by large language models generate visualizations and natural-language explanations from open-ended queries. Evaluating these multimodal outputs is challenging: curated reference benchmarks are costly to author, cannot comprehensively capture the space of valid responses, and are unavailable in production. Building on the Lexara evaluation framework, we introduce Lexara-RF, a reference-free set of metrics that scores CVA outputs using only the prompt, data, and model response. We reformulate evaluation as verification: 13 metrics operationalize visualization design theory and Gricean cooperative principles as computable consistency, intent-alignment, and design validity checks. On a human-rated corpus of CVA test-cases, Lexara-RF achieves alignment comparable to reference-based formulations, outperforms surface-similarity NLG baselines, and localizes structurally grounded failures with high accuracy.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use. Builds on 16 of its 38 references that are in the map. First posted on arXiv 15 Sep 2026.
Builds on
16 of 38 references.
References · 16
- 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
- 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
- 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
- 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
- 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
- 2024VisEval: A Benchmark for Data Visualization in the Era of Large Language Modelscited by 45
- 2021Collecting and Characterizing Natural Language Utterances for Specifying Data Visualizationscited by 31
- 2022How do you Converse with an Analytical Chatbot? Revisiting Gricean Maxims for Designing Analytical Conversational Behaviorcited by 20
- 2024Vi(E)va LLM! A Conceptual Stack for Evaluating and Interpreting Generative AI-based Visualizationscited by 10
- 2023Marrying Dialogue Systems with Data Visualization: Interactive Data Visualization Generation from Natural Language Conversationscited by 9
4 more
- 2025VIS-Shepherd: Constructing Critic for LLM-based Data Visualization Generationcited by 9
- 2025SimVecVis: A Dataset for Enhancing MLLMs in Visualization Understandingcited by 5
- 2025Evalet: Evaluating Large Language Models through Functional Fragmentationcited by 3
- 2026Lexara: A User-Centered Toolkit for Evaluating Large Language Models for Conversational Visual Analyticscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Can AI agents understand spoken conversations about data visualizations in online meetings?
- 2025Text2Vis: A Challenging and Diverse Benchmark for Generating Multimodal Visualizations from Text
- 2026InSight: A Benchmark for Agentic Claim Verification in Interactive Visualizations
- 2026VegaChat: A Robust Framework for LLM-Based Chart Generation and Assessment
- 2026Operationalising Multi-Dimensional Evaluation for Conversational Agents: A Scalable, Governed Pipeline with Selective Re-evaluation and Model Benchmarking
- 2025InterChat: Enhancing Generative Visual Analytics using Multimodal Interactions
- 2025See or Recall: A Sanity Check for the Role of Vision in Solving Visualization Question Answer Tasks with Multimodal LLMs
- 2025Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models