mlsanityData to 7 Oct 2026

Domain-specific Continued Pretraining of Language Models for Capturing Long Context in Mental Health

arXiv 2304.10447 ↗First posted 20 Apr 2023cs.CLcited by 25

Not on the map. Why?

Abstract (arXiv)

Pretrained language models have been used in various natural language processing applications. In the mental health domain, domain-specific language models are pretrained and released, which facilitates the early detection of mental health conditions. Social posts, e.g., on Reddit, are usually long documents. However, there are no domain-specific pretrained models for long-sequence modeling in the mental health domain. This paper conducts domain-specific continued pretraining to capture the long context for mental health. Specifically, we train and release MentalXLNet and MentalLongformer based on XLNet and Longformer. We evaluate the mental health classification performance and the long-range ability of these two domain-specific pretrained models. Our models are released in HuggingFace.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 18 of its 35 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

18 of 35 references.

References · 18

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2013Crowdsourcing a Word-Emotion Association Lexiconcited by 193
  10. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 159
  11. 2019Dreaddit: A Reddit Dataset for Stress Analysis in Social Mediacited by 81
  12. 2023Will Affective Computing Emerge from Foundation Models and General AI? A First Evaluation on ChatGPTcited by 41
6 more
  1. 2022CAMS: An Annotated Corpus for Causal Analysis of Mental Health Issues in Social Media Postscited by 34
  2. 2019Suicidal Ideation Detection: A Review of Machine Learning Methods and Applicationscited by 31
  3. 2019Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilationcited by 30
  4. 2020Suicidal Ideation and Mental Disorder Detection with Attentive Relation Networkscited by 28
  5. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  6. 2023Emotion fusion for mental illness detection from social media: A surveycited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 10; 2025: 5; 2026: 4.

25 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 162
  2. 2024SaulLM-7B: A pioneering Large Language Model for Lawcited by 136
  3. 2023MentaLLaMA: Interpretable Mental Health Analysis on Social Media with Large Language Modelscited by 63
  4. 2024ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notescited by 22
  5. 2024Depression Detection on Social Media with Large Language Modelscited by 20
  6. 2023Rethinking Large Language Models in Mental Health Applicationscited by 16
  7. 2024Diverse Perspectives, Divergent Models: Cross-Cultural Evaluation of Depression Detection on Twittercited by 12
  8. 2024Still Not Quite There! Evaluating Large Language Models for Comorbid Mental Health Diagnosiscited by 12
  9. 2024Pre-training Differentially Private Models with Limited Public Datacited by 8
  10. 2024On-Policy Self-Alignment with Fine-grained Knowledge Feedback for Hallucination Mitigationcited by 8
  11. 2025MHQA: A Diverse, Knowledge Intensive Mental Health Question Answering Challenge for Language Modelscited by 7
  12. 2023Multimodal Foundation Models For Echocardiogram Interpretationcited by 6
13 more
  1. 2024On the State of NLP Approaches to Modeling Depression in Social Media: A Post-COVID-19 Outlookcited by 6
  2. 2025ProMind-LLM: Proactive Mental Health Care via Causal Reasoning with Sensor Datacited by 6
  3. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
  4. 2024FAIIR: Building Toward A Conversational AI Agent Assistant for Youth Mental Health Service Provisioncited by 3
  5. 2025Fluent but Unfeeling: The Emotional Blind Spots of Language Modelscited by 2
  6. 2023Conceptualizing Suicidal Behavior: Utilizing Explanations of Predicted Outcomes to Analyze Longitudinal Social Media Datacited by 1
  7. 2024Building Multilingual Datasets for Predicting Mental Health Severity through LLMs: Prospects and Challengescited by 1
  8. 2025Fine-Tuning LLMs for Report Summarization: Analysis on Supervised and Unsupervised Datacited by 1
  9. 2026OMIND: Framework for Knowledge Grounded Finetuning and Multi-Turn Dialogue Benchmark for Mental Health LLMscited by 1
  10. 2023Cordyceps@LT-EDI: Depression Detection with Reddit and Self-trainingcited by 0
  11. 2026FedMental: Evaluating Federated Learning for Mental Health Detection from Social Media Datacited by 0
  12. 2026Snugi-AI-v2 @ eRisk 2026 Task 2: Early Depression Detection via a Learned Stopping Policy with Sustained Confidence Gatecited by 0
  13. 2026BiGraph-Diffuse: A Bidirectional Diffusion Language Model with Graph-Structured Retrieval For Mental Health Counselingcited by 0