Identifying & Interactively Refining Ambiguous User Goals for Data Visualization Code Generation
Not on the map. Why?
Abstract (arXiv)
Establishing shared goals is a fundamental step in human-AI communication. However, ambiguities can lead to outputs that seem correct but fail to reflect the speaker's intent. In this paper, we explore this issue with a focus on the data visualization domain, where ambiguities in natural language impact the generation of code that visualizes data. The availability of multiple views on the contextual (e.g., the intended plot and the code rendering the plot) allows for a unique and comprehensive analysis of diverse ambiguity types. We develop a taxonomy of types of ambiguity that arise in this task and propose metrics to quantify them. Using Matplotlib problems from the DS-1000 dataset, we demonstrate that our ambiguity metrics better correlate with human annotations than uncertainty baselines. Our work also explores how multi-turn dialogue can reduce ambiguity, therefore, improve code accuracy by better matching user goals. We evaluate three pragmatic models to inform our dialogue strategies: Gricean Cooperativity, Discourse Representation Theory, and Questions under Discussion. A simulated user study reveals how pragmatic dialogues reduce ambiguity and enhance code accuracy, highlighting the value of multi-turn exchanges in code generation.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 78 references that are in the map. First posted on arXiv 10 Oct 2025.
Builds on
33 of 78 references.
References · 33
- 2021Evaluating Large Language Models Trained on Codecited by 8,053
- 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
- 2023Code Llama: Open Foundation Models for Codecited by 2,105
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
- 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
- 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
- 2023We're Afraid Language Models Aren't Modeling Ambiguitycited by 124
- 2023Prompting and Evaluating Large Language Models for Proactive Dialogues: Clarification, Target-guided, and Non-collaborationcited by 115
- 2024STaR-GATE: Teaching Language Models to Ask Clarifying Questionscited by 99
- 2024MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualizationcited by 98
- 2022Coder Reviewer Reranking for Code Generationcited by 83
21 more
- 2020NL4DV: A Toolkit for Generating Analytic Specifications for Data Visualization from Natural Language Queriescited by 73
- 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 66
- 2017Colors in Context: A Pragmatic Neural Model for Grounded Language Understandingcited by 57
- 2022What is it like to program with artificial intelligence?cited by 39
- 2023Benchmarking and Improving Text-to-SQL Generation under Ambiguitycited by 31
- 2024How Beginning Programmers and Code LLMs (Mis)read Each Othercited by 31
- 2024The RealHumanEval: Evaluating Large Language Models' Abilities to Support Programmerscited by 28
- 2020Clue: Cross-modal Coherence Modeling for Caption Generationcited by 24
- 2023Is AI the better programming partner? Human-Human Pair Programming vs. Human-AI pAIr Programmingcited by 20
- 2024Accounting for Sycophancy in Language Model Uncertainty Estimationcited by 18
- 2023On the Reliability and Explainability of Language Models for Program Generationcited by 17
- 2022Python Code Generation by Asking Clarification Questionscited by 16
- 2020Program Synthesis with Pragmatic Communicationcited by 15
- 2022Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussioncited by 13
- 2024AmbigNLG: Addressing Task Ambiguity in Instruction for NLGcited by 10
- 2022Towards Unification of Discourse Annotation Frameworkscited by 8
- 2024Do Pre-Trained Language Models Detect and Understand Semantic Underspecification? Ask the DUST!cited by 7
- 2023Generating Pragmatic Examples to Train Neural Program Synthesizerscited by 5
- 2024Picturing Ambiguity: A Visual Twist on the Winograd Schema Challengecited by 3
- 2024Amortizing Pragmatic Program Synthesis with Rankingscited by 3
- 2023Model Analysis & Evaluation for Ambiguous Question Answeringcited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.