A Multimodal Conversational Agent for Tabular Data Analysis
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Large language models (LLMs) can reshape information processing by handling data analysis, visualization, and interpretation in an interactive, context-aware dialogue with users, including voice interaction, while maintaining high performance. In this article, we present Talk2Data, a multimodal LLM-driven conversational agent for intuitive data exploration. The system lets users query datasets with voice or text instructions and receive answers as plots, tables, statistics, or spoken explanations. Built on LLMs, the suggested design combines OpenAI Whisper automatic speech recognition (ASR) system, Qwen-coder code generation LLM/model, custom sandboxed execution tools, and Coqui library for text-to-speech (TTS) within an agentic orchestration loop. Unlike text-only analysis tools, it adapts responses across modalities and supports multi-turn dialogues grounded in dataset context. In an evaluation of 48 tasks on three datasets, our prototype achieved 95.8% accuracy with model-only generation time under 1.7 seconds (excluding ASR and execution time). A comparison across five LLM sizes (1.5B-32B) revealed accuracy-latency-cost trade-offs, with a 7B model providing the best balance for interactive use. By routing between conversation with user and code execution, constrained to a transparent sandbox, with simultaneously grounding prompts in schema-level context, the Talk2Data agent reliably retrieves actionable insights from tables while making computations verifiable. In the article, except for the Talk2Data agent itself, we discuss implications for human-data interaction, trust in LLM-driven analytics, and future extensions toward large-scale multimodal assistants.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 14 of its 29 references that are in the map. First posted on arXiv 23 Nov 2025.
Builds on
14 of 29 references.
References · 14
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2023Code Llama: Open Foundation Models for Codecited by 2,139
- 2024Qwen2.5-Coder Technical Reportcited by 1,203
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 561
- 2018Neural Approaches to Conversational AIcited by 165
- 2023Information Retrieval: Recent Advances and Beyondcited by 49
- 2024Multimodal Table Understandingcited by 44
- 2023Natural Language Interfaces for Tabular Data Querying and Visualization: A Surveycited by 27
- 2023cTBLS: Augmenting Large Language Models with Conversational Tablescited by 7
- 2024LLM4DS: Evaluating Large Language Models for Data Science Code Generationcited by 7
- 2023QTSumm: Query-Focused Summarization over Tabular Datacited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025MMAT-1M: A Large Reasoning Dataset for Multimodal Agent Tuning
- 2025Spoken Conversational Agents with Large Language Models
- 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
- 2024InsightLens: Augmenting LLM-Powered Data Analysis with Interactive Insight Management and Navigation
- 2025LLM/Agent-as-Data-Analyst: A Survey
- 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Survey
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agents
- 2025ConDABench: Interactive Evaluation of Language Models for Data Analysis