mlsanityData to 7 Oct 2026

Large Language Models Perform on Par with Experts Identifying Mental Health Factors in Adolescent Online Forums

arXiv 2404.16461 ↗First posted 25 Apr 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Mental health in children and adolescents has been steadily deteriorating over the past few years. The recent advent of Large Language Models (LLMs) offers much hope for cost and time efficient scaling of monitoring and intervention, yet despite specifically prevalent issues such as school bullying and eating disorders, previous studies on have not investigated performance in this domain or for open information extraction where the set of answers is not predetermined. We create a new dataset of Reddit posts from adolescents aged 12-19 annotated by expert psychiatrists for the following categories: TRAUMA, PRECARITY, CONDITION, SYMPTOMS, SUICIDALITY and TREATMENT and compare expert labels to annotations from two top performing LLMs (GPT3.5 and GPT4). In addition, we create two synthetic datasets to assess whether LLMs perform better when annotating data as they generate it. We find GPT4 to be on par with human inter-annotator agreement and performance on synthetic data to be substantially higher, however we find the model still occasionally errs on issues of negation and factuality and higher performance on synthetic data is driven by greater complexity of real data rather than inherent advantage.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 15 of its 39 references that are in the map. First posted on arXiv 25 Apr 2024.

Builds on

15 of 39 references.

References · 15

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2019What BERT is not: Lessons from a new suite of psycholinguistic diagnostics for language modelscited by 317
  8. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  9. 2022Large Language Models are Few-Shot Clinical Information Extractorscited by 206
  10. 2023DeID-GPT: Zero-shot Medical Text De-Identification by GPT-4cited by 128
  11. 2023Mental-LLM: Leveraging Large Language Models for Mental Health Prediction via Online Text Datacited by 66
  12. 2023Towards Interpretable Mental Health Analysis with Large Language Modelscited by 53
3 more
  1. 2022Clinical Prompt Learning with Frozen Language Modelscited by 19
  2. 2024Efficiency at Scale: Investigating the Performance of Diminutive Language Models in Clinical Taskscited by 4
  3. 2024Detecting the Clinical Features of Difficult-to-Treat Depression using Synthetic Data from Large Language Modelscited by 2

Built on by

Not cited yet.