# Bridging the LLM Accessibility Divide? Performance, Fairness, and Cost of Closed versus Open LLMs for Automated Essay Scoring

arXiv: https://arxiv.org/abs/2503.11827 · HTML: https://mlsanity.com/paper/2503.11827/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 50 references that are in the map.
- First posted on arXiv 14 Mar 2025.

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing](https://mlsanity.com/paper/2001.00973/) (2020)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)
- [Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: An Empirical Study](https://mlsanity.com/paper/2304.00723/) (2023)
- [Text Summarization Using Large Language Models: A Comparative Study of MPT-7b-instruct, Falcon-7b-instruct, and OpenAI Chat-GPT Models](https://mlsanity.com/paper/2310.10449/) (2023)
- [Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production](https://mlsanity.com/paper/2312.14972/) (2023)
- [Exploring the Relationship between In-Context Learning and Instruction Tuning](https://mlsanity.com/paper/2311.10367/) (2023)
- [When Automated Assessment Meets Automated Content Generation: Examining Text Quality in the Era of GPTs](https://mlsanity.com/paper/2309.14488/) (2023)
- [Risks and Opportunities of Open-Source Generative AI](https://mlsanity.com/paper/2405.08597/) (2024)
- [Comparing GPT-4 and Open-Source Language Models in Misinformation Mitigation](https://mlsanity.com/paper/2401.06920/) (2024)
- [Unraveling the Capabilities of Language Models in News Summarization](https://mlsanity.com/paper/2501.18128/) (2025)

## Built on by (most cited first, 7 of 7)

- [A Multi-Task Evaluation of LLMs' Processing of Academic Text Input](https://mlsanity.com/paper/2508.11779/) (2025)
- [The Tower of Babel Revisited: Multilingual Jailbreak Prompts on Closed-Source Large Language Models](https://mlsanity.com/paper/2505.12287/) (2025)
- [Agreement Between Large Language Models and Human Raters in Essay Scoring: A Research Synthesis](https://mlsanity.com/paper/2512.14561/) (2025)
- [Large Language Models for IT Automation Tasks: Are We There Yet?](https://mlsanity.com/paper/2505.20505/) (2025)
- [Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays](https://mlsanity.com/paper/2602.04604/) (2026)
- [Who Annotates in NLP? A Large-scale Assessment of Human Annotation Reporting between 2018 and 2025](https://mlsanity.com/paper/2606.02255/) (2026)
- [From Feature-Based Models to Generative AI: Validity Evidence for Constructed Response Scoring](https://mlsanity.com/paper/2603.19280/) (2026)

## Abstract (arXiv)

Closed large language models (LLMs) such as GPT-4 have set state-of-the-art results across a number of NLP tasks and have become central to NLP and machine learning (ML)-driven solutions. Closed LLMs' performance and wide adoption has sparked considerable debate about their accessibility in terms of availability, cost, and transparency. In this study, we perform a rigorous comparative analysis of nine leading LLMs, spanning closed, open, and open-source LLM ecosystems, across text assessment and generation tasks related to automated essay scoring. Our findings reveal that for few-shot learning-based assessment of human generated essays, open LLMs such as Llama 3 and Qwen2.5 perform comparably to GPT-4 in terms of predictive performance, with no significant differences in disparate impact scores when considering age- or race-related fairness. Moreover, Llama 3 offers a substantial cost advantage, being up to 37 times more cost-efficient than GPT-4. For generative tasks, we find that essays generated by top open LLMs are comparable to closed LLMs in terms of their semantic composition/embeddings and ML assessed scores. Our findings challenge the dominance of closed LLMs and highlight the democratizing potential of open LLMs, suggesting they can effectively bridge accessibility divides while maintaining competitive performance and fairness.