# Predicting challenge moments from students' discourse: A comparison of GPT-4 to two traditional natural language processing approaches

arXiv: https://arxiv.org/abs/2401.01692 · HTML: https://mlsanity.com/paper/2401.01692/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 59 references that are in the map.
- First posted on arXiv 3 Jan 2024.

## Builds on (most cited first, 4 of 4)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [GoEmotions: A Dataset of Fine-Grained Emotions](https://mlsanity.com/paper/2005.00547/) (2020)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)

## Built on by (most cited first, 8 of 8)

- [Large Language Models for Education: A Survey and Outlook](https://mlsanity.com/paper/2403.18105/) (2024)
- [Confabulation: The Surprising Value of Large Language Model Hallucinations](https://mlsanity.com/paper/2406.04175/) (2024)
- [The Interplay of Learning, Analytics, and Artificial Intelligence in Education: A Vision for Hybrid Intelligence](https://mlsanity.com/paper/2403.16081/) (2024)
- [Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis](https://mlsanity.com/paper/2405.03677/) (2024)
- [Creating Artificial Students that Never Existed: Leveraging Large Language Models and CTGANs for Synthetic Data Generation](https://mlsanity.com/paper/2501.01793/) (2025)
- [Rethinking the Potential of Multimodality in Collaborative Problem Solving Diagnosis with Large Language Models](https://mlsanity.com/paper/2504.15093/) (2025)
- [Multimodal Analytics of Cybersecurity Crisis Preparation Exercises: What Predicts Success?](https://mlsanity.com/paper/2603.28553/) (2026)
- [Walking through Discussions: A Mobile Visual Analytics System for In-Situ Group Discussion Analysis](https://mlsanity.com/paper/2608.08617/) (2026)

## Abstract (arXiv)

Effective collaboration requires groups to strategically regulate themselves to overcome challenges. Research has shown that groups may fail to regulate due to differences in members' perceptions of challenges which may benefit from external support. In this study, we investigated the potential of leveraging three distinct natural language processing models: an expert knowledge rule-based model, a supervised machine learning (ML) model and a Large Language model (LLM), in challenge detection and challenge dimension identification (cognitive, metacognitive, emotional and technical/other challenges) from student discourse, was investigated. The results show that the supervised ML and the LLM approaches performed considerably well in both tasks, in contrast to the rule-based approach, whose efficacy heavily relies on the engineered features by experts. The paper provides an extensive discussion of the three approaches' performance for automated detection and support of students' challenge moments in collaborative learning activities. It argues that, although LLMs provide many advantages, they are unlikely to be the panacea to issues of the detection and feedback provision of socially shared regulation of learning due to their lack of reliability, as well as issues of validity evaluation, privacy and confabulation. We conclude the paper with a discussion on additional considerations, including model transparency to explore feasible and meaningful analytical feedback for students and educators using LLMs.