# Using Generative Text Models to Create Qualitative Codebooks for Student Evaluations of Teaching

arXiv: https://arxiv.org/abs/2403.11984 · HTML: https://mlsanity.com/paper/2403.11984/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 104 references that are in the map.
- First posted on arXiv 18 Mar 2024.

## Builds on (most cited first, 19 of 19)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [MTEB: Massive Text Embedding Benchmark](https://mlsanity.com/paper/2210.07316/) (2022)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Natural Language Processing: State of The Art, Current Trends and Challenges](https://mlsanity.com/paper/1708.05148/) (2017)
- [AnglE-optimized Text Embeddings](https://mlsanity.com/paper/2309.12871/) (2023)
- [Supporting Qualitative Analysis with Large Language Models: Combining Codebook with GPT-3 for Deductive Coding](https://mlsanity.com/paper/2304.10548/) (2023)
- [Distributional Semantics and Linguistic Theory](https://mlsanity.com/paper/1905.01896/) (2019)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [A Transformer-based approach to Irony and Sarcasm detection](https://mlsanity.com/paper/1911.10401/) (2019)
- [A Review of the Trends and Challenges in Adopting Natural Language Processing Methods for Education Feedback Analysis](https://mlsanity.com/paper/2301.08826/) (2023)
- [SIGHT: A Large Annotated Dataset on Student Insights Gathered from Higher Education Transcripts](https://mlsanity.com/paper/2306.09343/) (2023)
- [The Utility of Large Language Models and Generative AI for Education Research](https://mlsanity.com/paper/2305.18125/) (2023)
- [Exploring the Efficacy of ChatGPT in Analyzing Student Teamwork Feedback with an Existing Taxonomy](https://mlsanity.com/paper/2305.11882/) (2023)

## Built on by (most cited first, 4 of 4)

- [Thematic Analysis with Open-Source Generative AI and Machine Learning: A New Method for Inductive Qualitative Codebook Development](https://mlsanity.com/paper/2410.03721/) (2024)
- [Automatic deductive coding in discourse analysis: an application of large language models in learning analytics](https://mlsanity.com/paper/2410.01240/) (2024)
- [Leveraging Large Language Models for Actionable Course Evaluation Student Feedback to Lecturers](https://mlsanity.com/paper/2407.01274/) (2024)
- [Teaching at Scale: Leveraging AI to Evaluate and Elevate Engineering Education](https://mlsanity.com/paper/2508.02731/) (2025)

## Abstract (arXiv)

Feedback is a critical aspect of improvement. Unfortunately, when there is a lot of feedback from multiple sources, it can be difficult to distill the information into actionable insights. Consider student evaluations of teaching (SETs), which are important sources of feedback for educators. They can give instructors insights into what worked during a semester. A collection of SETs can also be useful to administrators as signals for courses or entire programs. However, on a large scale as in high-enrollment courses or administrative records over several years, the volume of SETs can render them difficult to analyze. In this paper, we discuss a novel method for analyzing SETs using natural language processing (NLP) and large language models (LLMs). We demonstrate the method by applying it to a corpus of 5,000 SETs from a large public university. We show that the method can be used to extract, embed, cluster, and summarize the SETs to identify the themes they express. More generally, this work illustrates how to use the combination of NLP techniques and LLMs to generate a codebook for SETs. We conclude by discussing the implications of this method for analyzing SETs and other types of student writing in teaching and research settings.