# Can ChatGPT and Bard Generate Aligned Assessment Items? A Reliability Analysis against Human Performance

arXiv: https://arxiv.org/abs/2304.05372 · HTML: https://mlsanity.com/paper/2304.05372/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 0 of its 19 references that are in the map.
- First posted on arXiv 9 Apr 2023.

## Built on by (most cited first, 4 of 4)

- [Aegis:An Advanced LLM-Based Multi-Agent for Intelligent Functional Safety Engineering](https://mlsanity.com/paper/2410.12475/) (2024)
- [EALM: Introducing Multidimensional Ethical Alignment in Conversational Information Retrieval](https://mlsanity.com/paper/2310.00970/) (2023)
- [Analysis of the User Perception of Chatbots in Education Using A Partial Least Squares Structural Equation Modeling Approach](https://mlsanity.com/paper/2311.03636/) (2023)
- [Can GPT replace human raters? Validity and reliability of machine-generated norms for metaphors](https://mlsanity.com/paper/2512.12444/) (2025)

## Abstract (arXiv)

ChatGPT and Bard are AI chatbots based on Large Language Models (LLM) that are slated to promise different applications in diverse areas. In education, these AI technologies have been tested for applications in assessment and teaching. In assessment, AI has long been used in automated essay scoring and automated item generation. One psychometric property that these tools must have to assist or replace humans in assessment is high reliability in terms of agreement between AI scores and human raters. In this paper, we measure the reliability of OpenAI ChatGP and Google Bard LLMs tools against experienced and trained humans in perceiving and rating the complexity of writing prompts. Intraclass correlation (ICC) as a performance metric showed that the inter-reliability of both the OpenAI ChatGPT and the Google Bard were low against the gold standard of human ratings.