# Leveraging Prompts in LLMs to Overcome Imbalances in Complex Educational Text Data

arXiv: https://arxiv.org/abs/2407.01551 · HTML: https://mlsanity.com/paper/2407.01551/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 49 references that are in the map.
- First posted on arXiv 28 Apr 2024.

## Builds on (most cited first, 10 of 10)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Imbalanced-learn: A Python Toolbox to Tackle the Curse of Imbalanced Datasets in Machine Learning](https://mlsanity.com/paper/1609.06570/) (2016)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Prompts Matter: Insights and Strategies for Prompt Engineering in Automated Software Traceability](https://mlsanity.com/paper/2308.00229/) (2023)
- [Foundations of data imbalance and solutions for a data democracy](https://mlsanity.com/paper/2108.00071/) (2021)
- [Investigating the Learning Behaviour of In-context Learning: A Comparison with Supervised Learning](https://mlsanity.com/paper/2307.15411/) (2023)
- [Efficient Classification of Student Help Requests in Programming Courses Using Large Language Models](https://mlsanity.com/paper/2310.20105/) (2023)
- [Assertion Enhanced Few-Shot Learning: Instructive Technique for Large Language Models to Generate Educational Explanations](https://mlsanity.com/paper/2312.03122/) (2023)

## Abstract (arXiv)

In this paper, we explore the potential of Large Language Models (LLMs) with assertions to mitigate imbalances in educational datasets. Traditional models often fall short in such contexts, particularly due to the complexity and nuanced nature of the data. This issue is especially prominent in the education sector, where cognitive engagement levels among students show significant variation in their open responses. To test our hypothesis, we utilized an existing technology for assertion-based prompt engineering through an 'Iterative - ICL PE Design Process' comparing traditional Machine Learning (ML) models against LLMs augmented with assertions (N=135). Further, we conduct a sensitivity analysis on a subset (n=27), examining the variance in model performance concerning classification metrics and cognitive engagement levels in each iteration. Our findings reveal that LLMs with assertions significantly outperform traditional ML models, particularly in cognitive engagement levels with minority representation, registering up to a 32% increase in F1-score. Additionally, our sensitivity study indicates that incorporating targeted assertions into the LLM tested on the subset enhances its performance by 11.94%. This improvement primarily addresses errors stemming from the model's limitations in understanding context and resolving lexical ambiguities in student responses.