# Evaluating the Effectiveness of the Foundational Models for Q&A Classification in Mental Health care arXiv: https://arxiv.org/abs/2406.15966 ยท HTML: https://mlsanity.com/paper/2406.15966/ ## Key facts - Cited by 1 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 64 references that are in the map. - First posted on arXiv 23 Jun 2024. ## Builds on (most cited first, 30 of 32) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019) - [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019) - [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019) - [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018) - [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019) - [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020) - [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019) - [AraBERT: Transformer-based Model for Arabic Language Understanding](https://mlsanity.com/paper/2003.00104/) (2020) - [ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic](https://mlsanity.com/paper/2101.01785/) (2020) - [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023) - [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021) - [The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models](https://mlsanity.com/paper/2103.06678/) (2021) - [Using millions of emoji occurrences to learn any-domain representations for detecting sentiment, emotion and sarcasm](https://mlsanity.com/paper/1708.00524/) (2017) - [PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Support](https://mlsanity.com/paper/2106.01702/) (2021) - [Semi-supervised sequence tagging with bidirectional language models](https://mlsanity.com/paper/1705.00108/) (2017) - [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019) - [Human-AI Collaboration Enables More Empathic Conversations in Text-based Peer-to-Peer Mental Health Support](https://mlsanity.com/paper/2203.15144/) (2022) - [ChatCounselor: A Large Language Models for Mental Health Support](https://mlsanity.com/paper/2309.15461/) (2023) - [SpanEmo: Casting Multi-label Emotion Classification as Span-prediction](https://mlsanity.com/paper/2101.10038/) (2021) - [CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Posts](https://mlsanity.com/paper/2207.04674/) (2022) - [Arabic natural language processing: An overview](https://mlsanity.com/paper/1903.02784/) (2019) - [LAraBench: Benchmarking Arabic AI with Large Language Models](https://mlsanity.com/paper/2305.14982/) (2023) - [Large Language Model for Mental Health: A Systematic Review](https://mlsanity.com/paper/2403.15401/) (2024) - [Emotion fusion for mental illness detection from social media: A survey](https://mlsanity.com/paper/2304.09493/) (2023) - [Deep Learning Mental Health Dialogue System](https://mlsanity.com/paper/2301.09412/) (2023) - [EmoMent: An Emotion Annotated Mental Health Corpus from two South Asian Countries](https://mlsanity.com/paper/2208.08486/) (2022) - [A Comparison of SVM against Pre-trained Language Models (PLMs) for Text Classification Tasks](https://mlsanity.com/paper/2211.02563/) (2022) - [WellXplain: Wellness Concept Extraction and Classification in Reddit Posts for Mental Health Analysis](https://mlsanity.com/paper/2308.13710/) (2023) - [MentalQA: An Annotated Arabic Corpus for Questions and Answers of Mental Healthcare](https://mlsanity.com/paper/2405.12619/) (2024) ## Built on by (most cited first, 1 of 1) - [CLIN-LLM: A Safety-Constrained Hybrid Framework for Clinical Diagnosis and Treatment Generation](https://mlsanity.com/paper/2510.22609/) (2025) ## Abstract (arXiv) Pre-trained Language Models (PLMs) have the potential to transform mental health support by providing accessible and culturally sensitive resources. However, despite this potential, their effectiveness in mental health care and specifically for the Arabic language has not been extensively explored. To bridge this gap, this study evaluates the effectiveness of foundational models for classification of Questions and Answers (Q&A) in the domain of mental health care. We leverage the MentalQA dataset, an Arabic collection featuring Q&A interactions related to mental health. In this study, we conducted experiments using four different types of learning approaches: traditional feature extraction, PLMs as feature extractors, Fine-tuning PLMs and prompting large language models (GPT-3.5 and GPT-4) in zero-shot and few-shot learning settings. While traditional feature extractors combined with Support Vector Machines (SVM) showed promising performance, PLMs exhibited even better results due to their ability to capture semantic meaning. For example, MARBERT achieved the highest performance with a Jaccard Score of 0.80 for question classification and a Jaccard Score of 0.86 for answer classification. We further conducted an in-depth analysis including examining the effects of fine-tuning versus non-fine-tuning, the impact of varying data size, and conducting error analysis. Our analysis demonstrates that fine-tuning proved to be beneficial for enhancing the performance of PLMs, and the size of the training data played a crucial role in achieving high performance. We also explored prompting, where few-shot learning with GPT-3.5 yielded promising results. There was an improvement of 12% for question and classification and 45% for answer classification. Based on our findings, it can be concluded that PLMs and prompt-based approaches hold promise for mental health support in Arabic.