# Enhancing Essay Scoring with Adversarial Weights Perturbation and Metric-specific AttentionPooling

arXiv: https://arxiv.org/abs/2401.05433 · HTML: https://mlsanity.com/paper/2401.05433/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 18 references that are in the map.
- First posted on arXiv 6 Jan 2024.

## Builds on (most cited first, 2 of 2)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharing](https://mlsanity.com/paper/2111.09543/) (2021)

## Built on by (most cited first, 5 of 5)

- [Text Understanding and Generation Using Transformer Models for Intelligent E-commerce Recommendations](https://mlsanity.com/paper/2402.16035/) (2024)
- [Comprehensive Implementation of TextCNN for Enhanced Collaboration between Natural Language Processing and System Recommendation](https://mlsanity.com/paper/2403.09718/) (2024)
- [Advancing Prompt Recovery in NLP: A Deep Dive into the Integration of Gemma-2b-it and Phi2 Models](https://mlsanity.com/paper/2407.05233/) (2024)
- [Utilizing Deep Learning for Enhancing Network Resilience in Finance](https://mlsanity.com/paper/2402.09820/) (2024)
- [Emerging Synergies Between Large Language Models and Machine Learning in Ecommerce Recommendations](https://mlsanity.com/paper/2403.02760/) (2024)

## Abstract (arXiv)

The objective of this study is to improve automated feedback tools designed for English Language Learners (ELLs) through the utilization of data science techniques encompassing machine learning, natural language processing, and educational data analytics. Automated essay scoring (AES) research has made strides in evaluating written essays, but it often overlooks the specific needs of English Language Learners (ELLs) in language development. This study explores the application of BERT-related techniques to enhance the assessment of ELLs' writing proficiency within AES. To address the specific needs of ELLs, we propose the use of DeBERTa, a state-of-the-art neural language model, for improving automated feedback tools. DeBERTa, pretrained on large text corpora using self-supervised learning, learns universal language representations adaptable to various natural language understanding tasks. The model incorporates several innovative techniques, including adversarial training through Adversarial Weights Perturbation (AWP) and Metric-specific AttentionPooling (6 kinds of AP) for each label in the competition. The primary focus of this research is to investigate the impact of hyperparameters, particularly the adversarial learning rate, on the performance of the model. By fine-tuning the hyperparameter tuning process, including the influence of 6AP and AWP, the resulting models can provide more accurate evaluations of language proficiency and support tailored learning tasks for ELLs. This work has the potential to significantly benefit ELLs by improving their English language proficiency and facilitating their educational journey.