# Multi-Level Feedback Generation with Large Language Models for Empowering Novice Peer Counselors

arXiv: https://arxiv.org/abs/2403.15482 · HTML: https://mlsanity.com/paper/2403.15482/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 64 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Builds on (most cited first, 18 of 18)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [Towards Emotional Support Dialog Systems](https://mlsanity.com/paper/2106.01144/) (2021)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [Can large language models provide useful feedback on research papers? A large-scale empirical analysis](https://mlsanity.com/paper/2310.01783/) (2023)
- [A Computational Approach to Understanding Empathy Expressed in Text-Based Mental Health Support](https://mlsanity.com/paper/2009.08441/) (2020)
- [CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large Language Models for Data Annotation](https://mlsanity.com/paper/2310.15638/) (2023)
- [Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization](https://mlsanity.com/paper/2010.01672/) (2020)
- [A Computational Framework for Behavioral Assessment of LLM Therapists](https://mlsanity.com/paper/2401.00820/) (2024)
- [Modeling Motivational Interviewing Strategies On An Online Peer-to-Peer Counseling Platform](https://mlsanity.com/paper/2211.05182/) (2022)
- [Boosting Distress Support Dialogue Responses with Motivational Interviewing Strategy](https://mlsanity.com/paper/2305.10195/) (2023)
- [Automated Quality Assessment of Cognitive Behavioral Therapy Sessions Through Highly Contextualized Language Representations](https://mlsanity.com/paper/2102.11573/) (2021)
- [VERVE: Template-based ReflectiVE Rewriting for MotiVational IntErviewing](https://mlsanity.com/paper/2311.08299/) (2023)
- [What Makes Digital Support Effective? How Therapeutic Skills Affect Clinical Well-Being](https://mlsanity.com/paper/2312.10775/) (2023)

## Built on by (most cited first, 16 of 16)

- ['Simulacrum of Stories': Examining Large Language Models as Qualitative Research Participants](https://mlsanity.com/paper/2409.19430/) (2024)
- [A Survey of Large Language Models in Psychotherapy: Current Landscape and Future Directions](https://mlsanity.com/paper/2502.11095/) (2025)
- [Linguistic Comparison of AI- and Human-Written Responses to Online Mental Health Queries](https://mlsanity.com/paper/2504.09271/) (2025)
- [Large Language Model Agents for Improving Engagement with Behavior Change Interventions: Application to Digital Mindfulness](https://mlsanity.com/paper/2407.13067/) (2024)
- [TN-Eval: Rubric and Evaluation Protocols for Measuring the Quality of Behavioral Therapy Notes](https://mlsanity.com/paper/2503.20648/) (2025)
- [Responsible Evaluation of AI for Mental Health](https://mlsanity.com/paper/2602.00065/) (2026)
- [AI for Proactive Mental Health: A Multi-Institutional, Longitudinal, Randomized Controlled Trial](https://mlsanity.com/paper/2601.11530/) (2025)
- ["Is This Really a Human Peer Supporter?": Misalignments Between Peer Supporters and Experts in LLM-Supported Interactions](https://mlsanity.com/paper/2506.09354/) (2025)
- ["I Said Things I Needed to Hear Myself": Peer Support as an Emotional, Organisational, and Sociotechnical Practice in Singapore](https://mlsanity.com/paper/2506.09362/) (2025)
- [First, Do No Harm: AI Supervisor Scaffolds Novice Growth in Counselor Education](https://mlsanity.com/paper/2508.09042/) (2025)
- [Controlled Yet Natural: A Hybrid BDI-LLM Conversational Agent for Child Helpline Training](https://mlsanity.com/paper/2509.16784/) (2025)
- [Multi-dimensional Assessment and Explainable Feedback for Counselor Responses to Client Resistance in Text-based Counseling with LLMs](https://mlsanity.com/paper/2602.21638/) (2026)
- [CounselReflect: Opportunities and Challenges for Designing Tools to Support Self-Reflection on Mental Health and Well-Being Conversations with AI](https://mlsanity.com/paper/2603.29429/) (2026)
- [Adapting from Downturns: Prediction of Long-Term Conversational-Skill Development in Mental-Health Crisis Counselors](https://mlsanity.com/paper/2609.04350/) (2026)
- [PeerPen: AI-Assisted Writing for Online Mental Health Peer Support](https://mlsanity.com/paper/2609.14886/) (2026)
- [From Pattern Recognizers to Personalized Companions: A Survey of Large Language Models in Mental Health](https://mlsanity.com/paper/2609.25186/) (2026)

## Abstract (arXiv)

Realistic practice and tailored feedback are key processes for training peer counselors with clinical skills. However, existing mechanisms of providing feedback largely rely on human supervision. Peer counselors often lack mechanisms to receive detailed feedback from experienced mentors, making it difficult for them to support the large number of people with mental health issues who use peer counseling. Our work aims to leverage large language models to provide contextualized and multi-level feedback to empower peer counselors, especially novices, at scale. To achieve this, we co-design with a group of senior psychotherapy supervisors to develop a multi-level feedback taxonomy, and then construct a publicly available dataset with comprehensive feedback annotations of 400 emotional support conversations. We further design a self-improvement method on top of large language models to enhance the automatic generation of feedback. Via qualitative and quantitative evaluation with domain experts, we demonstrate that our method minimizes the risk of potentially harmful and low-quality feedback generation which is desirable in such high-stakes scenarios.