mlsanityData to 7 Oct 2026

MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare

arXiv 2110.15621 ↗First posted 29 Oct 2021cs.CLcited by 159

Not on the map. Why?

Abstract (arXiv)

Mental health is a critical issue in modern society, and mental disorders could sometimes turn to suicidal ideation without adequate treatment. Early detection of mental disorders and suicidal ideation from social content provides a potential way for effective social intervention. Recent advances in pretrained contextualized language representations have promoted the development of several domain-specific pretrained models and facilitated several downstream applications. However, there are no existing pretrained language models for mental healthcare. This paper trains and release two pretrained masked language models, i.e., MentalBERT and MentalRoBERTa, to benefit machine learning for the mental healthcare research community. Besides, we evaluate our trained domain-specific models and several variants of pretrained language models on several mental disorder detection benchmarks and demonstrate that language representations pretrained in the target domain improve the performance of mental health detection tasks.

Key facts. Cited by 159 papers in the map (data through 7 Oct 2026). Builds on 17 of its 50 references that are in the map. First posted on arXiv 29 Oct 2021.

Builds on

17 of 50 references.

References · 17

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  5. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  6. 2019Publicly Available Clinical BERT Embeddingscited by 995
  7. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  8. 2019How to Fine-Tune BERT for Text Classification?cited by 409
  9. 2019Dreaddit: A Reddit Dataset for Stress Analysis in Social Mediacited by 81
  10. 2019Fine-tuning Pre-Trained Transformer Language Models to Distantly Supervised Relation Extractioncited by 36
  11. 2019Suicidal Ideation Detection: A Review of Machine Learning Methods and Applicationscited by 31
  12. 2019BERT-based Ranking for Biomedical Entity Normalizationcited by 29
5 more
  1. 2019Recent Trends in Deep Learning Based Personality Detectioncited by 29
  2. 2020Suicidal Ideation and Mental Disorder Detection with Attentive Relation Networkscited by 28
  3. 2021Early Risk Detection of Pathological Gambling, Self-Harm and Depression Using BERTcited by 16
  4. 2020Utilizing BERT Intermediate Layers for Aspect Based Sentiment Analysis and Natural Language Inferencecited by 14
  5. 2021Micromodels for Efficient, Explainable, and Reusable Systems: A Case Study on Mental Healthcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 36; 2024: 48; 2025: 36; 2026: 30.

159 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  3. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  4. 2023MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Modelscited by 63
  5. 2024The opportunities and risks of large language models in mental healthcited by 54
  6. 2023Towards Interpretable Mental Health Analysis with Large Language Modelscited by 53
  7. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  8. 2025MentalChat16K: A Benchmark Dataset for Conversational Mental Health Assistancecited by 38
  9. 2023Training Models to Generate, Recognize, and Reframe Unhelpful Thoughtscited by 31
  10. 2023Empowering Psychotherapy with Large Language Models: Cognitive Distortion Detection through Diagnosis of Thought Promptingcited by 31
  11. 2022A Unified Review of Deep Learning for Automated Medical Codingcited by 26
  12. 2023Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Healthcited by 25
38 more
  1. 2022Hierarchical Attention Network for Explainable Depression Detection on Twitter Aided by Metaphor Concept Mappingscited by 23
  2. 2023Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyondcited by 22
  3. 2022Symptom Identification for Interpretable Detection of Multiple Mental Disorderscited by 20
  4. 2024Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contextscited by 19
  5. 2024PATIENT-Psi: Using Large Language Models to Simulate Patients for Training Mental Health Professionalscited by 19
  6. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  7. 2022Gendered Mental Health Stigma in Masked Language Modelscited by 17
  8. 2023An Annotated Dataset for Explainable Interpersonal Risk Factors of Mental Disturbance in Social Media Postscited by 16
  9. 2023Rethinking Large Language Models in Mental Health Applicationscited by 16
  10. 2024Understanding the Effects of Iterative Prompting on Truthfulnesscited by 14
  11. 2023Towards Healthy AI: Large Language Models Need Therapists Toocited by 13
  12. 2023Emotion fusion for mental illness detection from social media: A surveycited by 13
  13. 2023Values, Ethics, Morals? On the Use of Moral Concepts in NLP Researchcited by 13
  14. 2023Calibration of Transformer-based Models for Identifying Stress and Depression in Social Mediacited by 12
  15. 2024Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twittercited by 12
  16. 2024Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosiscited by 12
  17. 2024Chinese MentalBERT: Domain-Adaptive Pre-training on Social Media for Chinese Mental Health Text Analysiscited by 11
  18. 2024Few-shot Dialogue Strategy Learning for Motivational Interviewing via Inductive Reasoningcited by 11
  19. 2024WellDunn: On the Robustness and Explainability of Language Models and Large Language Models in Identifying Wellness Dimensionscited by 10
  20. 2025Are LLMs effective psychological assessors? Leveraging adaptive RAG for interpretable mental health screening through psychometric practicecited by 10
  21. 2023ChatGPT for Suicide Risk Assessment on Social Media: Quantitative Evaluation of Model Performance, Potentials and Limitationscited by 9
  22. 2023Explainable Depression Symptom Detection in Social Mediacited by 9
  23. 2024MentalGLM Series: Explainable Large Language Models for Mental Health Analysis on Chinese Social Mediacited by 9
  24. 2023A Cross Attention Approach to Diagnostic Explainability using Clinical Practice Guidelines for Depressioncited by 8
  25. 2025FedMentalCare: Towards Privacy-Preserving Fine-Tuned LLMs to Analyze Mental Health Status Using Federated Learning Frameworkcited by 8
  26. 2022Benchmarking for Public Health Surveillance tasks on Social Media with a Domain-Specific Pretrained Language Modelcited by 7
  27. 2024A Novel Nuanced Conversation Evaluation Framework for Large Language Models in Mental Healthcited by 7
  28. 2024A Comprehensive Evaluation of Large Language Models on Mental Illnessescited by 7
  29. 2024KidLM: Advancing Language Models for Children -- Early Insights and Future Directionscited by 7
  30. 2025MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Modelscited by 7
  31. 2024Explainable AI for Mental Disorder Detection via Social Media: A survey and outlookcited by 6
  32. 2024On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlookcited by 6
  33. 2025Figurative-cum-Commonsense Knowledge Infusion for Multimodal Mental Health Meme Classificationcited by 6
  34. 2025Survey and Experiments on Mental Disorder Detection via Social Media: From Large Language Models and RAG to Agentscited by 6
  35. 2025ProMind-LLM: Proactive Mental Health Care via Causal Reasoning with Sensor Datacited by 6
  36. 2025PsychiatryBench: A Multi-Task Benchmark for LLMs in Psychiatrycited by 6
  37. 2022Modern Views of Machine Learning for Precision Psychiatrycited by 5
  38. 2023Robust language-based mental health assessments in time and space through social mediacited by 5

50 most cited of 159 All 159 (JSON)