mlsanityData to 7 Oct 2026

Tapilot-Crossing: Benchmarking and Evolving LLMs Towards Interactive Data Analysis Agents

arXiv 2403.05307 ↗First posted 8 Mar 2024cs.AIcited by 17

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Interactive Data Analysis, the collaboration between humans and LLM agents, enables real-time data exploration for informed decision-making. The challenges and costs of collecting realistic interactive logs for data analysis hinder the quantitative evaluation of Large Language Model (LLM) agents in this task. To mitigate this issue, we introduce Tapilot-Crossing, a new benchmark to evaluate LLM agents on interactive data analysis. Tapilot-Crossing contains 1024 interactions, covering 4 practical scenarios: Normal, Action, Private, and Private Action. Notably, Tapilot-Crossing is constructed by an economical multi-agent environment, Decision Company, with few human efforts. We evaluate popular and advanced LLM agents in Tapilot-Crossing, which underscores the challenges of interactive data analysis. Furthermore, we propose Adaptive Interaction Reflection (AIR), a self-generated reflection strategy that guides LLM agents to learn from successful history. Experiments demonstrate that Air can evolve LLMs into effective interactive data analysis agents, achieving a relative performance improvement of up to 44.5%.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 44 of its 67 references that are in the map. First posted on arXiv 8 Mar 2024.

Builds on

44 of 67 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments22 Feb
  2. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMs19 Oct
  3. 2023OpenAgents: An Open Platform for Language Agents in the Wild16 Oct
  4. 2023Lemur: Harmonizing Natural Language and Code for Language Agents10 Oct
  5. 2023SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents22 May
  6. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar

Other references · 38

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2018Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Taskcited by 1,120
  7. 2023Teaching Large Language Models to Self-Debugcited by 963
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  9. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  10. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  11. 2023API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMscited by 574
  12. 2023DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correctioncited by 465
26 more
  1. 2022DS-1000: A Natural and Reliable Benchmark for Data Science Code Generationcited by 442
  2. 2023Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluationcited by 378
  3. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  4. 2023MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedbackcited by 293
  5. 2024OpenCodeInterpreter: Integrating Code Generation with Execution and Refinementcited by 242
  6. 2023MetaTool Benchmark for Large Language Models: Deciding Whether to Use Tools and Which to Usecited by 209
  7. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  8. 2024CodeS: Towards Building Open-source Language Models for Text-to-SQLcited by 183
  9. 2019CoSQL: A Conversational Text-to-SQL Challenge Towards Cross-Domain Natural Language Interfaces to Databasescited by 180
  10. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
  11. 2019SParC: Cross-Domain Semantic Parsing in Contextcited by 141
  12. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  13. 2023PRD: Peer Rank and Discussion Improve Large Language Model based Evaluationscited by 114
  14. 2022Natural Language to Code Generation in Interactive Data Science Notebookscited by 86
  15. 2023Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflowcited by 76
  16. 2023ReAcTable: Enhancing ReAct for Table Question Answeringcited by 76
  17. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  18. 2022When Language Model Meets Private Librarycited by 54
  19. 2023Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioningcited by 47
  20. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  21. 2023Text2Analysis: A Benchmark of Table Question Answering with Advanced Data Analysis and Unclear Queriescited by 38
  22. 2023LLM-Assisted Code Cleaning For Training Accurate Code Generatorscited by 27
  23. 2020An Imitation Game for Learning Semantic Parsers from User Interactioncited by 23
  24. 2023Learning to Simulate Natural Language Feedback for Interactive Semantic Parsingcited by 19
  25. 2023Beyond Generating Code: Evaluating GPT on a Data Visualization Coursecited by 16
  26. 2023Iterative Forward Tuning Boosts In-Context Learning in Language Modelscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 8; 2026: 6.

17 papers.

Sort:

By area:

  1. 2024SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulationcited by 100
  2. 2024BLADE: Benchmarking Language Model Agents for Data-Driven Sciencecited by 72
  3. 2024Aviary: training language agents on challenging scientific taskscited by 23
  4. 2025Large Language Model-based Data Science Agent: A Surveycited by 22
  5. 2025Self-Evolving Multi-Agent Systems via Textual Backpropagationcited by 21
  6. 2025BIRD-INTERACT: Re-imagining Text-to-SQL Evaluation for Large Language Models via Lens of Dynamic Interactionscited by 17
  7. 2025SHARE: An SLM-based Hierarchical Action CorREction Assistant for Text-to-SQLcited by 15
  8. 2025Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflowscited by 9
  9. 2026Autonomous Research Agents: A Survey of AI Scientists and the Verification Gapcited by 6
  10. 2025Measuring Data Science Automation: A Survey of Evaluation Tools for AI Assistants and Agentscited by 4
  11. 2026LongDS-Bench: On the Failure of Long-Horizon Agentic Data Analysiscited by 4
  12. 2025ConDABench: Interactive Evaluation of Language Models for Data Analysiscited by 3
5 more
  1. 2025Micro-Act: Mitigating Knowledge Conflict in LLM-based RAG via Actionable Self-Reasoningcited by 1
  2. 2026Generating Expressive and Customizable Evals for Timeseries Data Analysis Agents with AgentFuelcited by 1
  3. 2026Fisher-R1: Training LLM Agents for Reliable Hypothesis Testingcited by 0
  4. 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysiscited by 0
  5. 2026TrustDABench: Benchmarking Reliability and Robustness of LLMs for Structured Data Analysiscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025IDA-Bench: Evaluating LLMs on Interactive Guided Data Analysis
  2. 2025EDGE: Efficient Data Selection for LLM Agents via Guideline Effectiveness
  3. 2024Benchmark Self-Evolving: A Multi-Agent Framework for Dynamic LLM Evaluation
  4. 2023AgentBench: Evaluating LLMs as Agents
  5. 2024A Survey on Large Language Model-based Agents for Statistics and Data Science
  6. 2025Multi-Mission Tool Bench: Assessing the Robustness of LLM based Agents through Related and Dynamic Missions
  7. 2025LLM-Based Data Science Agents: A Survey of Capabilities, Challenges, and Future Directions
  8. 2026InteractBench: Benchmarking LLMs on Competitive Programming under Unrevealed Information