# COACH meets QUORUM: A Framework and Pipeline for Aligning User, Expert and Developer Perspectives in LLM-generated Health Counselling

arXiv: https://arxiv.org/abs/2603.08392 · HTML: https://mlsanity.com/paper/2603.08392/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 11 of its 36 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Builds on (most cited first, 11 of 11)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Survey](https://mlsanity.com/paper/2402.17944/) (2024)
- [Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks](https://mlsanity.com/paper/2410.20266/) (2024)
- [Prompt Engineering for Healthcare: Methodologies and Applications](https://mlsanity.com/paper/2304.14670/) (2023)
- [Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs](https://mlsanity.com/paper/2310.10358/) (2023)
- [Understanding User Experience in Large Language Model Interactions](https://mlsanity.com/paper/2401.08329/) (2024)
- [GPTCoach: Towards LLM-Based Physical Activity Coaching](https://mlsanity.com/paper/2405.06061/) (2024)
- [Virtual Agents for Alcohol Use Counseling: Exploring LLM-Powered Motivational Interviewing](https://mlsanity.com/paper/2407.08095/) (2024)
- [Fool Me, Fool Me: User Attitudes Toward LLM Falsehoods](https://mlsanity.com/paper/2412.11625/) (2024)
- [When LLMs Can't Help: Real-World Evaluation of LLMs in Nutrition](https://mlsanity.com/paper/2511.20652/) (2025)

## Abstract (arXiv)

Systems that collect data on sleep, mood, and activities can provide valuable lifestyle counselling to populations affected by chronic disease and its consequences. Such systems are, however, challenging to develop; besides reliably extracting patterns from user-specific data, systems should also contextualise these patterns with validated medical knowledge to ensure the quality of counselling, and generate counselling that is relevant to a real user. We present QUORUM, a new evaluation framework that unifies these developer-, expert-, and user-centric perspectives, and show with a real case study that it meaningfully tracks convergence and divergence in stakeholder perspectives. We also present COACH, a Large Language Model-driven pipeline to generate personalised lifestyle counselling for our Healthy Chronos use case, a diary app for cancer patients and survivors. Applying our framework shows that overall, users, medical experts, and developers converge on the opinion that the generated counselling is relevant, of good quality, and reliable. However, stakeholders also diverge on the tone of the counselling, sensitivity to errors in pattern-extraction, and potential hallucinations. These findings highlight the importance of multi-stakeholder evaluation for consumer health language technologies and illustrate how a unified evaluation framework can support trustworthy, patient-centered NLP systems in real-world settings.