mlsanityData to 7 Oct 2026

mhGPT: A Lightweight Generative Pre-Trained Transformer for Mental Health Text Analysis

arXiv 2408.08261 ↗First posted 15 Aug 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

This paper introduces mhGPT, a lightweight generative pre-trained transformer trained on mental health-related social media and PubMed articles. Fine-tuned for specific mental health tasks, mhGPT was evaluated under limited hardware constraints and compared with state-of-the-art models like MentaLLaMA and Gemma. Despite having only 1.98 billion parameters and using just 5% of the dataset, mhGPT outperformed larger models and matched the performance of models trained on significantly more data. The key contributions include integrating diverse mental health data, creating a custom tokenizer, and optimizing a smaller architecture for low-resource settings. This research could advance AI-driven mental health care, especially in areas with limited computing power.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 21 of its 50 references that are in the map. First posted on arXiv 15 Aug 2024.

Builds on

21 of 50 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  11. 2022Large Language Models Can Self-Improvecited by 656
  12. 2021Datasets: A Community Library for Natural Language Processingcited by 526
9 more
  1. 2018An Empirical Model of Large-Batch Trainingcited by 268
  2. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 159
  3. 2023NEFTune: Noisy Embeddings Improve Instruction Finetuningcited by 99
  4. 2019Dreaddit: A Reddit Dataset for Stress Analysis in Social Mediacited by 81
  5. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 65
  6. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  7. 2023An Annotated Dataset for Explainable Interpersonal Risk Factors of Mental Disturbance in Social Media Postscited by 16
  8. 2021Natural Language Processing for Smart Healthcarecited by 14
  9. 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025The Rise of Small Language Models in Healthcare: A Comprehensive Surveycited by 18
  2. 2024CBT-Bench: Evaluating Large Language Models on Assisting Cognitive Behavior Therapycited by 12
  3. 2025Menta: A Small Language Model for On-Device Mental Health Predictioncited by 3
  4. 2026From Token to Token Pair: Efficient Prompt Compression for Large Language Models in Clinical Predictioncited by 2
  5. 2025Evaluating Small Language Models for Agentic On-Farm Decision Support Systemscited by 0