# CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback

arXiv: https://arxiv.org/abs/2504.02323 · HTML: https://mlsanity.com/paper/2504.02323/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 76 references that are in the map.
- First posted on arXiv 3 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models](https://mlsanity.com/paper/2404.18796/) (2024)
- [Chain-of-Verification Reduces Hallucination in Large Language Models](https://mlsanity.com/paper/2309.11495/) (2023)
- [Active Prompting with Chain-of-Thought for Large Language Models](https://mlsanity.com/paper/2302.12246/) (2023)
- [Applying Large Language Models and Chain-of-Thought for Automatic Scoring](https://mlsanity.com/paper/2312.03748/) (2023)
- [Students' Voices on Generative AI: Perceptions, Benefits, and Challenges in Higher Education](https://mlsanity.com/paper/2305.00290/) (2023)
- [A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science](https://mlsanity.com/paper/2403.14565/) (2024)
- [A large language model-assisted education tool to provide feedback on open-ended responses](https://mlsanity.com/paper/2308.02439/) (2023)
- [Matching Exemplar as Next Sentence Prediction (MeNSP): Zero-shot Prompt Learning for Automatic Scoring in Science Education](https://mlsanity.com/paper/2301.08771/) (2023)
- [Using Large Language Models for Automated Grading of Student Writing about Science](https://mlsanity.com/paper/2412.18719/) (2024)
- [Large Language Models for Education: Grading Open-Ended Questions Using ChatGPT](https://mlsanity.com/paper/2307.16696/) (2023)
- [Using ChatGPT for Science Learning: A Study on Pre-service Teachers' Lesson Planning](https://mlsanity.com/paper/2402.01674/) (2024)
- [Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis](https://mlsanity.com/paper/2405.03677/) (2024)
- [Grading Massive Open Online Courses Using Large Language Models](https://mlsanity.com/paper/2406.11102/) (2024)
- [Giving Feedback on Interactive Student Programs with Meta-Exploration](https://mlsanity.com/paper/2211.08802/) (2022)
- [H-AES: Towards Automated Essay Scoring for Hindi](https://mlsanity.com/paper/2302.14635/) (2023)
- [APE: Active Learning-based Tooling for Finding Informative Few-shot Examples for LLM-based Entity Matching](https://mlsanity.com/paper/2408.04637/) (2024)
- [Data Augmentation for Sparse Multidimensional Learning Performance Data Using Generative AI](https://mlsanity.com/paper/2409.15631/) (2024)
- [Reducing the Cost: Cross-Prompt Pre-Finetuning for Short Answer Scoring](https://mlsanity.com/paper/2408.13966/) (2024)

## Built on by (most cited first, 5 of 5)

- [Feedback by Design: Understanding and Overcoming User Feedback Barriers in Conversational Agents](https://mlsanity.com/paper/2602.01405/) (2026)
- [Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents](https://mlsanity.com/paper/2602.01415/) (2026)
- [Deep Interaction: An Efficient Human-AI Interaction Method for Large Reasoning Models](https://mlsanity.com/paper/2607.14049/) (2026)
- [Feedback Without the Wait: Piloting a Generative AI Practice Platform in a Large Maths Class](https://mlsanity.com/paper/2610.01262/) (2026)
- [Agreement Is Not Validity: Cross-Model LLM Consensus in Diagnosing Student Failure Modes in K-12 Math Tutoring Dialogue](https://mlsanity.com/paper/2610.08703/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have created new opportunities to assist teachers and support student learning. While researchers have explored various prompt engineering approaches in educational contexts, the degree to which these approaches generalize across domains--such as science, computing, and engineering--remains underexplored. In this paper, we introduce Chain-of-Thought Prompting + Active Learning (CoTAL), an LLM-based approach to formative assessment scoring that (1) leverages Evidence-Centered Design (ECD) to align assessments and rubrics with curriculum goals, (2) applies human-in-the-loop prompt engineering to automate response scoring, and (3) incorporates chain-of-thought (CoT) prompting and teacher and student feedback to iteratively refine questions, rubrics, and LLM prompts. Our findings demonstrate that CoTAL improves GPT-4's scoring performance across domains, achieving gains of up to 38.9% over a non-prompt-engineered baseline (i.e., without labeled examples, chain-of-thought prompting, or iterative refinement). Teachers and students judge CoTAL to be effective at scoring and explaining responses, and their feedback produces valuable insights that enhance grading accuracy and explanation quality.