mlsanityData to 7 Oct 2026

ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing

arXiv 2309.09128 ↗First posted 17 Sep 2023cs.HC cs.AIcited by 88

Not on the map. Why?

Abstract (arXiv)

Evaluating outputs of large language models (LLMs) is challenging, requiring making -- and making sense of -- many responses. Yet tools that go beyond basic prompting tend to require knowledge of programming APIs, focus on narrow domains, or are closed-source. We present ChainForge, an open-source visual toolkit for prompt engineering and on-demand hypothesis testing of text generation LLMs. ChainForge provides a graphical interface for comparison of responses across models and prompt variations. Our system was designed to support three tasks: model selection, prompt template design, and hypothesis testing (e.g., auditing). We released ChainForge early in its development and iterated on its design with academics and online users. Through in-lab and interview studies, we find that a range of people could use ChainForge to investigate hypotheses that matter to them, including in real-world settings. We identify three modes of prompt engineering and LLM hypothesis testing: opportunistic exploration, limited evaluation, and iterative refinement.

Key facts. Cited by 88 papers in the map (data through 7 Oct 2026). Builds on 17 of its 53 references that are in the map. First posted on arXiv 17 Sep 2023.

Builds on

17 of 53 references.

References · 17

  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  3. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  4. 2022Black-Box Tuning for Language-Model-as-a-Servicecited by 254
  5. 2022Grounded Copilot: How Programmers Interact with Code-Generating Modelscited by 216
  6. 2022PromptChainer: Chaining Large Language Model Prompts through Visual Programmingcited by 134
  7. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  8. 2023CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classescited by 71
  9. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  10. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  11. 2023Graphologue: Exploring Large Language Model Responses with Interactive Diagramscited by 65
  12. 2022Designing for Responsible Trust in AI Systems: A Communication Perspectivecited by 48
5 more
  1. 2023Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Modelscited by 48
  2. 2023Supporting Human-AI Collaboration in Auditing LLMs with LLMscited by 46
  3. 2023PromptAid: Prompt Exploration, Perturbation, Testing and Iteration using Visual Analytics for Large Language Modelscited by 19
  4. 2023Model Sketching: Centering Concepts in Early-Stage Machine Learning Model Designcited by 15
  5. 2023Prompt Space Optimizing Few-shot Reasoning Success with Large Language Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 27; 2025: 36; 2026: 24.

88 papers.

Sort:

By area:

  1. 2024Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferencescited by 142
  2. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Modelscited by 85
  3. 2025Interactive Debugging and Steering of Multi-Agent AI Systemscited by 68
  4. 2024LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editingcited by 47
  5. 2025Creating General User Models from Computer Usecited by 44
  6. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 43
  7. 2024Cocoa: Co-Planning and Co-Execution with AI Agentscited by 38
  8. 2024How Far Are We From AGI: Are LLMs All We Need?cited by 31
  9. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  10. 2024Supporting Sensemaking of Large Language Model Outputs at Scalecited by 26
  11. 2024TeachTune: Reviewing Pedagogical Agents Against Diverse Student Profiles with Simulated Studentscited by 25
  12. 2024What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Usecited by 24
38 more
  1. 2024Unified Parameter-Efficient Unlearning for LLMscited by 24
  2. 2024Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualizationcited by 23
  3. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  4. 2025Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absentcited by 15
  5. 2025VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failurescited by 14
  6. 2024Not Just Novelty: A Longitudinal Study on Utility and Customization of an AI Workflowcited by 13
  7. 2024Wordflow: Social Prompt Engineering for Large Language Modelscited by 9
  8. 2025SPHERE: An Evaluation Card for Human-AI Systemscited by 9
  9. 2024Understanding Large Language Model Behaviors through Interactive Counterfactual Generation and Analysiscited by 8
  10. 2024MetaScientist: A Human-AI Synergistic Framework for Automated Mechanical Metamaterial Designcited by 8
  11. 2025An Empirical Study on Prompt Compression for Large Language Modelscited by 8
  12. 2024Imagining a Future of Designing with AI: Dynamic Grounding, Constructive Negotiation, and Sustainable Motivationcited by 7
  13. 2024Exploring the Potential of Large Language Models for Improving Digital Forensic Investigation Efficiencycited by 7
  14. 2025OnGoal: Tracking and Visualizing Conversational Goals in Multi-Turn Dialogue with Large Language Modelscited by 7
  15. 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
  16. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  17. 2024Exploring Empty Spaces: Human-in-the-Loop Data Augmentationcited by 6
  18. 2025Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptionscited by 6
  19. 2025When Should Users Check? Modeling Confirmation Frequency in Multi-Step Agentic AI Taskscited by 6
  20. 2024Interrogating AI: Characterizing Emergent Playful Interactions with ChatGPTcited by 5
  21. 2024Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviorscited by 5
  22. 2025Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmapcited by 5
  23. 2024VernaCopter: Disambiguated Natural-Language-Driven Robot via Formal Specificationscited by 4
  24. 2025Bridging the AI Adoption Gap: Designing an Interactive Pedagogical Agent for Higher Education Instructorscited by 4
  25. 2025What's the Difference? Supporting Users in Identifying the Effects of Prompt and Model Changes Through Token Patternscited by 4
  26. 2025RAGTrace: Understanding and Refining Retrieval-Generation Dynamics in Retrieval-Augmented Generationcited by 4
  27. 2024Maximizing User Experience with LLMOps-Driven Personalized Recommendation Systemscited by 3
  28. 2025Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scalecited by 3
  29. 2025DxHF: Providing High-Quality Human Feedback for LLM Alignment via Interactive Decompositioncited by 3
  30. 2025Illuminating LLM Coding Agents: Visual Analytics for Deeper Understanding and Enhancementcited by 3
  31. 2025Evalet: Evaluating Large Language Models through Functional Fragmentationcited by 3
  32. 2026Does Structured Intent Representation Generalize? A Cross-Language, Cross-Model Empirical Study of 5W3H Promptingcited by 3
  33. 2023InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMscited by 2
  34. 2024Understanding the Dataset Practitioners Behind Large Language Model Developmentcited by 2
  35. 2025FeedQUAC: Quick Unobtrusive AI-Generated Commentarycited by 2
  36. 2025XGraphRAG: Interactive Visual Analysis for Graph-based Retrieval-Augmented Generationcited by 2
  37. 2025ALIGN: Prompt-based Attribute Alignment for Reliable, Responsible, and Personalized LLM-based Decision-Makingcited by 2
  38. 2025When the Domain Expert Has No Time and the LLM Developer Has No Clinical Expertise: Real-World Lessons from LLM Co-Design in a Safety-Net Hospitalcited by 2

50 most cited of 88 All 88 (JSON)