# Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health

arXiv: https://arxiv.org/abs/2304.10447 · HTML: https://mlsanity.com/paper/2304.10447/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 20 Apr 2023.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Crowdsourcing a Word-Emotion Association Lexicon](https://mlsanity.com/paper/1308.6297/) (2013)
- [MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcare](https://mlsanity.com/paper/2110.15621/) (2021)
- [Dreaddit: A Reddit Dataset for Stress Analysis in Social Media](https://mlsanity.com/paper/1911.00133/) (2019)
- [Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPT](https://mlsanity.com/paper/2303.03186/) (2023)
- [CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Posts](https://mlsanity.com/paper/2207.04674/) (2022)
- [Suicidal Ideation Detection: A Review of Machine Learning Methods and Applications](https://mlsanity.com/paper/1910.12611/) (2019)
- [Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation](https://mlsanity.com/paper/1912.11975/) (2019)
- [Suicidal Ideation and Mental Disorder Detection with Attentive Relation Networks](https://mlsanity.com/paper/2004.07601/) (2020)
- [The NLP Task Effectiveness of Long-Range Transformers](https://mlsanity.com/paper/2202.07856/) (2022)
- [Emotion fusion for mental illness detection from social media: A survey](https://mlsanity.com/paper/2304.09493/) (2023)

## Built on by (most cited first, 25 of 25)

- [A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics](https://mlsanity.com/paper/2310.05694/) (2023)
- [SaulLM-7B: A pioneering Large Language Model for Law](https://mlsanity.com/paper/2403.03883/) (2024)
- [MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Models](https://mlsanity.com/paper/2309.13567/) (2023)
- [ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes](https://mlsanity.com/paper/2403.05795/) (2024)
- [Depression Detection on Social Media with Large Language Models](https://mlsanity.com/paper/2403.10750/) (2024)
- [Rethinking Large Language Models in Mental Health Applications](https://mlsanity.com/paper/2311.11267/) (2023)
- [Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twitter](https://mlsanity.com/paper/2406.15362/) (2024)
- [Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosis](https://mlsanity.com/paper/2410.03908/) (2024)
- [On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigation](https://mlsanity.com/paper/2406.12221/) (2024)
- [Pre-training Differentially Private Models with Limited Public Data](https://mlsanity.com/paper/2402.18752/) (2024)
- [MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Models](https://mlsanity.com/paper/2502.15418/) (2025)
- [Multimodal Foundation Models For Echocardiogram Interpretation](https://mlsanity.com/paper/2308.15670/) (2023)
- [On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlook](https://mlsanity.com/paper/2410.08793/) (2024)
- [ProMind-LLM: Proactive Mental Health Care via Causal Reasoning with Sensor Data](https://mlsanity.com/paper/2505.14038/) (2025)
- [Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learning](https://mlsanity.com/paper/2509.03906/) (2025)
- [FAIIR: Building Toward A Conversational AI Agent Assistant for Youth Mental Health Service Provision](https://mlsanity.com/paper/2405.18553/) (2024)
- [Fluent but Unfeeling: The Emotional Blind Spots of Language Models](https://mlsanity.com/paper/2509.09593/) (2025)
- [Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Data](https://mlsanity.com/paper/2312.08299/) (2023)
- [Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challenges](https://mlsanity.com/paper/2409.17397/) (2024)
- [Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Data](https://mlsanity.com/paper/2503.10676/) (2025)
- [OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMs](https://mlsanity.com/paper/2603.25105/) (2026)
- [Cordyceps@LT-EDI: Depression Detection with Reddit and Self-training](https://mlsanity.com/paper/2310.01418/) (2023)
- [FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Data](https://mlsanity.com/paper/2605.18936/) (2026)
- [Snugi-AI-v2 @ eRisk 2026 Task 2: Early Depression Detection via a Learned Stopping Policy with Sustained Confidence Gate](https://mlsanity.com/paper/2609.08161/) (2026)
- [BiGraph-Diffuse: A Bidirectional Diffusion Language Model with Graph-Structured Retrieval For Mental Health Counseling](https://mlsanity.com/paper/2609.29519/) (2026)

## Abstract (arXiv)

Pretrained language models have been used in various natural language processing applications. In the mental health domain, domain-specific language models are pretrained and released, which facilitates the early detection of mental health conditions. Social posts, e.g., on Reddit, are usually long documents. However, there are no domain-specific pretrained models for long-sequence modeling in the mental health domain. This paper conducts domain-specific continued pretraining to capture the long context for mental health. Specifically, we train and release MentalXLNet and MentalLongformer based on XLNet and Longformer. We evaluate the mental health classification performance and the long-range ability of these two domain-specific pretrained models. Our models are released in HuggingFace.