mlsanityData to 7 Oct 2026

Automated essay scoring using efficient transformer-based language models

arXiv 2102.13136 ↗First posted 25 Feb 2021cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Automated Essay Scoring (AES) is a cross-disciplinary effort involving Education, Linguistics, and Natural Language Processing (NLP). The efficacy of an NLP model in AES tests it ability to evaluate long-term dependencies and extrapolate meaning even when text is poorly written. Large pretrained transformer-based language models have dominated the current state-of-the-art in many NLP tasks, however, the computational requirements of these models make them expensive to deploy in practice. The goal of this paper is to challenge the paradigm in NLP that bigger is better when it comes to AES. To do this, we evaluate the performance of several fine-tuned pretrained NLP models with a modest number of parameters on an AES dataset. By ensembling our models, we achieve excellent results with fewer parameters than most pretrained transformer-based models.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 28 of its 51 references that are in the map. First posted on arXiv 25 Feb 2021.

Builds on

28 of 51 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  12. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
16 more
  1. 2018Deep contextualized word representationscited by 2,285
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2020Rethinking Attention with Performerscited by 1,584
  6. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  7. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  8. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  10. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,024
  11. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  12. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 215
  13. 2016Automatic Text Scoring Using Neural Networkscited by 51
  14. 2019Language models and Automated Essay Scoringcited by 35
  15. 2019Single Headed Attention RNN: Stop Thinking With Your Headcited by 25
  16. 2018Neural Automated Essay Scoring and Coherence Modeling for Adversarially Crafted Inputcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 3; 2024: 3; 2025: 1; 2026: 3.

12 papers.

  1. 2023Fine-tuning ChatGPT for Automatic Scoringcited by 50
  2. 2022Short-answer scoring with ensembles of pretrained language modelscited by 8
  3. 2024Beyond Agreement: Diagnosing the Rationale Alignment of Automated Essay Scoring Methods based on Linguistically-informed Counterfactualscited by 8
  4. 2022Data Augmentation for Automated Essay Scoring using Transformer Modelscited by 3
  5. 2024The Rise of Artificial Intelligence in Educational Measurement: Opportunities and Ethical Challengescited by 3
  6. 2023Using Language Models to Detect Alarming Student Responsescited by 1
  7. 2024On the Suitability of pre-trained foundational LLMs for Analysis in German Legal Educationcited by 1
  8. 2023Using language models in the implicit automated assessment of mathematical short answer itemscited by 0
  9. 2025Comparing Human and AI Rater Effects Using the Many-Facet Rasch Modelcited by 0
  10. 2026From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoringcited by 0
  11. 2026The Order Matters: Sequential Fine-Tuning of LLaMA for Coherent Automated Essay Scoringcited by 0
  12. 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessmentcited by 0