# PromptHive: Bringing Subject Matter Experts Back to the Forefront with Collaborative Prompt Engineering for Educational Content Creation

arXiv: https://arxiv.org/abs/2410.16547 · HTML: https://mlsanity.com/paper/2410.16547/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 66 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Builds on (most cited first, 18 of 18)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [The Butterfly Effect of Altering Prompts: How Small Changes and Jailbreaks Affect Large Language Model Performance](https://mlsanity.com/paper/2401.03729/) (2024)
- [The Metacognitive Demands and Opportunities of Generative AI](https://mlsanity.com/paper/2312.10893/) (2023)
- [Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models](https://mlsanity.com/paper/2304.09337/) (2023)
- [Large-scale Text-to-Image Generation Models for Visual Artists' Creative Works](https://mlsanity.com/paper/2210.08477/) (2022)
- [Generative Students: Using LLM-Simulated Student Profiles to Support Question Item Evaluation](https://mlsanity.com/paper/2405.11591/) (2024)
- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)
- [Leveraging LLM-Respondents for Item Evaluation: a Psychometric Analysis](https://mlsanity.com/paper/2407.10899/) (2024)
- [From Explainable to Interactive AI: A Literature Review on Current Trends in Human-AI Interaction](https://mlsanity.com/paper/2405.15051/) (2024)
- [Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods](https://mlsanity.com/paper/2307.08161/) (2023)
- [Evaluating ChatGPT's Decimal Skills and Feedback Generation in a Digital Learning Game](https://mlsanity.com/paper/2306.16639/) (2023)
- [ABScribe: Rapid Exploration & Organization of Multiple Writing Variations in Human-AI Co-Writing Tasks using Large Language Models](https://mlsanity.com/paper/2310.00117/) (2023)
- [One Prompt is not Enough: Automated Construction of a Mixture-of-Expert Prompts](https://mlsanity.com/paper/2407.00256/) (2024)
- [Exposing the Achilles' Heel: Evaluating LLMs Ability to Handle Mistakes in Mathematical Reasoning](https://mlsanity.com/paper/2406.10834/) (2024)
- [Wordflow: Social Prompt Engineering for Large Language Models](https://mlsanity.com/paper/2401.14447/) (2024)
- [It Cannot Be Right If It Was Written by AI: On Lawyers' Preferences of Documents Perceived as Authored by an LLM vs a Human](https://mlsanity.com/paper/2407.06798/) (2024)

## Built on by (most cited first, 9 of 9)

- [Temperature and Persona Shape LLM Agent Consensus With Minimal Accuracy Gains in Qualitative Coding](https://mlsanity.com/paper/2507.11198/) (2025)
- [When the Domain Expert Has No Time and the LLM Developer Has No Clinical Expertise: Real-World Lessons from LLM Co-Design in a Safety-Net Hospital](https://mlsanity.com/paper/2508.08504/) (2025)
- [The Consensus Trap: Dissecting Subjectivity and the "Ground Truth" Illusion in Data Annotation](https://mlsanity.com/paper/2602.11318/) (2026)
- [Understanding Prompt Programming Tasks and Questions](https://mlsanity.com/paper/2507.17264/) (2025)
- [Botender: Supporting Communities in Collaboratively Designing AI Agents through Case-Based Provocations](https://mlsanity.com/paper/2509.25492/) (2025)
- [Data-Prompt Co-Evolution: Growing Test Sets to Refine LLM Behavior](https://mlsanity.com/paper/2510.12728/) (2025)
- [Key Considerations for Domain Expert Involvement in LLM Design and Evaluation: An Ethnographic Study](https://mlsanity.com/paper/2602.14357/) (2026)
- [Towards Synergistic Teacher-AI Interactions with Generative Artificial Intelligence](https://mlsanity.com/paper/2511.19580/) (2025)
- [The RIGID Framework: Research-Integrated, Generative AI-Mediated Instructional Design](https://mlsanity.com/paper/2603.12781/) (2026)

## Abstract (arXiv)

Involving subject matter experts in prompt engineering can guide LLM outputs toward more helpful, accurate, and tailored content that meets the diverse needs of different domains. However, iterating towards effective prompts can be challenging without adequate interface support for systematic experimentation within specific task contexts. In this work, we introduce PromptHive, a collaborative interface for prompt authoring, designed to better connect domain knowledge with prompt engineering through features that encourage rapid iteration on prompt variations. We conducted an evaluation study with ten subject matter experts in math and validated our design through two collaborative prompt-writing sessions and a learning gain study with 358 learners. Our results elucidate the prompt iteration process and validate the tool's usability, enabling non-AI experts to craft prompts that generate content comparable to human-authored materials while reducing perceived cognitive load by half and shortening the authoring process from several months to just a few hours.