Developing and evaluating a chatbot to support maternal health care
Not on the map. Why?
Abstract (arXiv)
The ability to provide trustworthy maternal health information using phone-based chatbots can have a significant impact, particularly in low-resource settings where users have low health literacy and limited access to care. However, deploying such systems is technically challenging: user queries are short, underspecified, and code-mixed across languages, answers require regional context-specific grounding, and partial or missing symptom context makes safe routing decisions difficult. We present a chatbot for maternal health in India developed through a partnership between academic researchers, a health tech company, a public health nonprofit, and a hospital. The system combines (1) stage-aware triage, routing high-risk queries to expert templates, (2) hybrid retrieval over curated maternal/newborn guidelines, and (3) evidence-conditioned generation from an LLM. Our core contribution is an evaluation workflow for high-stakes deployment under limited expert supervision. Targeting both component-level and end-to-end testing, we introduce: (i) a labeled triage benchmark (N=150) achieving 86.7% emergency recall, explicitly reporting the missed-emergency vs. over-escalation trade-off; (ii) a synthetic multi-evidence retrieval benchmark (N=100) with chunk-level evidence labels; (iii) LLM-as-judge comparison on real queries (N=781) using clinician-codesigned criteria; and (iv) expert validation. Our findings show that trustworthy medical assistants in multilingual, noisy settings require defense-in-depth design paired with multi-method evaluation, rather than any single model and evaluation method choice.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 17 of its 52 references that are in the map. First posted on arXiv 13 Mar 2026.
Builds on
17 of 52 references.
References · 17
- 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
- 2022Large Language Models Encode Clinical Knowledgecited by 1,830
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,463
- 2024A Survey on LLM-as-a-Judgecited by 1,423
- 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
- 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 638
- 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
- 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
- 2023MedCPT: Contrastive Pre-trained Transformers with Large-scale PubMed Search Logs for Zero-shot Biomedical Information Retrievalcited by 185
- 2024CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklistscited by 94
- 2024LLM-Rubric: A Multidimensional, Calibrated Approach to Automated Evaluation of Natural Language Textscited by 87
- 2022An Analysis of Fusion Functions for Hybrid Retrievalcited by 51
5 more
- 2024Language Ranker: A Metric for Quantifying LLM Performance Across High and Low-Resource Languagescited by 38
- 2023NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMscited by 14
- 2024Towards Leveraging Large Language Models for Automated Medical Q&A Evaluationcited by 14
- 2024HEALTH-PARIKSHA: Assessing RAG Models for Health Chatbots in Real-World Multilingual Settingscited by 4
- 2025Hierarchical Divide-and-Conquer for Fine-Grained Alignment in LLM-Based Medical Evaluationcited by 2
Built on by
Citing papers per year: 2026: 2.
2 papers.