# Exploring The Design of Prompts For Applying GPT-3 based Chatbots: A Mental Wellbeing Case Study on Mechanical Turk

arXiv: https://arxiv.org/abs/2209.11344 · HTML: https://mlsanity.com/paper/2209.11344/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Sep 2022.

## Built on by (most cited first, 13 of 13)

- [Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers](https://mlsanity.com/paper/2504.18412/) (2025)
- [Large Language Models in Mental Health Care: a Scoping Review](https://mlsanity.com/paper/2401.02984/) (2024)
- [Personality testing of Large Language Models: Limited temporal stability, but highlighted prosociality](https://mlsanity.com/paper/2306.04308/) (2023)
- [Does GPT-4 surpass human performance in linguistic pragmatics?](https://mlsanity.com/paper/2312.09545/) (2023)
- [Large Language Model for Mental Health: A Systematic Review](https://mlsanity.com/paper/2403.15401/) (2024)
- [Conversational AI-Powered Design: ChatGPT as Designer, User, and Product](https://mlsanity.com/paper/2302.07406/) (2023)
- [Response Generation for Cognitive Behavioral Therapy with Large Language Models: Comparative Study with Socratic Questioning](https://mlsanity.com/paper/2401.15966/) (2024)
- [Using Adaptive Bandit Experiments to Increase and Investigate Engagement in Mental Health](https://mlsanity.com/paper/2310.18326/) (2023)
- [Large Language Model Agents for Improving Engagement with Behavior Change Interventions: Application to Digital Mindfulness](https://mlsanity.com/paper/2407.13067/) (2024)
- [Somatic in the East, Psychological in the West?: Investigating Clinically-Grounded Cross-Cultural Depression Symptom Expression in LLMs](https://mlsanity.com/paper/2508.03247/) (2025)
- [Designing Mixed-Initiative Video Games](https://mlsanity.com/paper/2307.03877/) (2023)
- [Making a Long Story Short in Conversation Modeling](https://mlsanity.com/paper/2402.00143/) (2024)
- ["She was useful, but a bit too optimistic": Augmenting Design with Interactive Virtual Personas](https://mlsanity.com/paper/2508.19463/) (2025)

## Abstract (arXiv)

Large-Language Models like GPT-3 have the potential to enable HCI designers and researchers to create more human-like and helpful chatbots for specific applications. But evaluating the feasibility of these chatbots and designing prompts that optimize GPT-3 for a specific task is challenging. We present a case study in tackling these questions, applying GPT-3 to a brief 5-minute chatbot that anyone can talk to better manage their mood. We report a randomized factorial experiment with 945 participants on Mechanical Turk that tests three dimensions of prompt design to initialize the chatbot (identity, intent, and behaviour), and present both quantitative and qualitative analyses of conversations and user perceptions of the chatbot. We hope other HCI designers and researchers can build on this case study, for other applications of GPT-3 based chatbots to specific tasks, and build on and extend the methods we use for prompt design, and evaluation of the prompt design.