mlsanityData to 7 Oct 2026

ChainBuddy: An AI Agent System for Generating LLM Pipelines

arXiv 2409.13588 ↗First posted 20 Sep 2024cs.HC cs.AIcited by 6

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As large language models (LLMs) advance, their potential applications have grown significantly. However, it remains difficult to evaluate LLM behavior on user-defined tasks and craft effective pipelines to do so. Many users struggle with where to start, often referred to as the "blank page problem." ChainBuddy, an AI workflow generation assistant built into the ChainForge platform, aims to tackle this issue. From a single prompt or chat, ChainBuddy generates a starter evaluative LLM pipeline in ChainForge aligned to the user's requirements. ChainBuddy offers a straightforward and user-friendly way to plan and evaluate LLM behavior and make the process less daunting and more accessible across a wide range of possible tasks and use cases. We report a within-subjects user study comparing ChainBuddy to the baseline interface. We find that when using AI assistance, participants reported a less demanding workload, felt more confident, and produced higher quality pipelines evaluating LLM behavior. However, we also uncover a mismatch between subjective and objective ratings of performance: participants rated their successfulness similarly across conditions, while independent experts rated participant workflows significantly higher with AI assistance. Drawing connections to the Dunning-Kruger effect, we draw design implications for the future of workflow generation assistants to mitigate the risk of over-reliance.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 72 references that are in the map. First posted on arXiv 20 Sep 2024.

Builds on

26 of 72 references.

References · 26

  1. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  2. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  3. 2020Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performancecited by 396
  4. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  5. 2023Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Modelscited by 204
  6. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  7. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  8. 2021To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-makingcited by 118
  9. 2024Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffoldingcited by 115
  10. 2023ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testingcited by 88
  11. 2023EvalLM: Interactive Evaluation of Large Language Model Prompts on User-Defined Criteriacited by 73
  12. 2024AutoEval Done Right: Using Synthetic Data for Model Evaluationcited by 53
14 more
  1. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasetscited by 36
  2. 2024DynaVis: Dynamically Synthesized UI Widgets for Visualization Editingcited by 34
  3. 2024Aligning Language Models with Demonstrated Feedbackcited by 32
  4. 2024Teola: Towards End-to-End Optimization of LLM-based Applicationscited by 32
  5. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  6. 2024Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decompositioncited by 29
  7. 2023DSPy Assertions: Computational Constraints for Self-Refining Language Model Pipelinescited by 27
  8. 2023Prompt Sapper: A LLM-Empowered Production Tool for Building AI Chainscited by 24
  9. 2019DeepLine: AutoML Tool for Pipelines Generation using Deep Reinforcement Learning and Hierarchical Actions Filteringcited by 21
  10. 2024SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelinescited by 19
  11. 2023Beyond ChatBots: ExploreLLM for Structured Thoughts and Personalized Model Responsescited by 18
  12. 2022Telling Stories from Computational Notebooks: AI-Assisted Presentation Slides Creation for Presenting Data Science Workcited by 15
  13. 2024Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivationcited by 7
  14. 2023Co-audit: tools to help humans double-check AI-generated contentcited by 5

Built on by

Cited by year

Citing papers per year: 2025: 6.

6 papers.

  1. 2025Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AIcited by 39
  2. 2025Auto-Slides: An Interactive Multi-Agent System for Creating and Customizing Research Presentationscited by 18
  3. 2025ALLOY: Generating Reusable Agent Workflows from User Demonstrationcited by 3
  4. 2025XAgen: An Explainability Tool for Identifying and Correcting Failures in Multi-Agent Workflowscited by 3
  5. 2025SHERPA: A Model-Driven Framework for Large Language Model Executioncited by 1
  6. 2025Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocationscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Designing Algorithms Empowered by Language Models: An Analytical Framework, Case Studies, and Insights
  2. 2024AutoFlow: Automated Workflow Generation for Large Language Model Agents
  3. 2024A Review of Prominent Paradigms for LLM-Based Agents: Tool Use (Including RAG), Planning, and Feedback Learning
  4. 2024Automated test generation to evaluate tool-augmented LLMs as conversational AI agents
  5. 2024AgentQuest: A Modular Benchmark Framework to Measure Progress and Improve LLM Agents
  6. 2025LMR-BENCH: Evaluating LLM Agent's Ability on Reproducing Language Modeling Research
  7. 2024Creating an LLM-based AI-agent: A high-level methodology towards enhancing LLMs with APIs
  8. 2023AgentBench: Evaluating LLMs as Agents