# A Chain-of-Thought Prompting Approach with LLMs for Evaluating Students' Formative Assessment Responses in Science

arXiv: https://arxiv.org/abs/2403.14565 · HTML: https://mlsanity.com/paper/2403.14565/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Placed in Mathematical reasoning, one of 17 directions in LLM reasoning.
- Builds on 11 of its 39 references that are in the map.
- First posted on arXiv 21 Mar 2024.

## Placed in

- LLM reasoning / [Mathematical reasoning](https://mlsanity.com/lr/mathematical-reasoning/)

## Builds on (most cited first, 11 of 11)

- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicity](https://mlsanity.com/paper/2301.12867/) (2023)
- [Assessing the Quality of Multiple-Choice Questions Using GPT-4 and Rule-Based Methods](https://mlsanity.com/paper/2307.08161/) (2023)
- [Does Informativeness Matter? Active Learning for Educational Dialogue Act Classification](https://mlsanity.com/paper/2304.05578/) (2023)
- [Giving Feedback on Interactive Student Programs with Meta-Exploration](https://mlsanity.com/paper/2211.08802/) (2022)
- [Using Sampling to Estimate and Improve Performance of Automated Scoring Systems with Guarantees](https://mlsanity.com/paper/2111.08906/) (2021)
- [H-AES: Towards Automated Essay Scoring for Hindi](https://mlsanity.com/paper/2302.14635/) (2023)

## Built on by (most cited first, 30 of 31)

- [EducationQ: Evaluating LLMs' Teaching Capabilities Through Multi-Agent Dialogue Framework](https://mlsanity.com/paper/2504.14928/) (2025)
- [A LLM-Powered Automatic Grading Framework with Human-Level Guidelines Optimization](https://mlsanity.com/paper/2410.02165/) (2024)
- [Scaffolding Empathy: Training Counselors with Simulated Patients and Utterance-level Performance Visualizations](https://mlsanity.com/paper/2502.18673/) (2025)
- [Reasoning with OmniThought: A Large CoT Dataset with Verbosity and Cognitive Difficulty Annotations](https://mlsanity.com/paper/2505.10937/) (2025)
- [Unveiling Scoring Processes: Dissecting the Differences between LLMs and Human Graders in Automatic Scoring](https://mlsanity.com/paper/2407.18328/) (2024)
- [Towards A Human-in-the-Loop LLM Approach to Collaborative Discourse Analysis](https://mlsanity.com/paper/2405.03677/) (2024)
- [A Theory of Adaptive Scaffolding for LLM-Based Pedagogical Agents](https://mlsanity.com/paper/2508.01503/) (2025)
- [Attack-in-the-Chain: Bootstrapping Large Language Models for Attacks Against Black-box Neural Ranking Models](https://mlsanity.com/paper/2412.18770/) (2024)
- [Personalizing Student-Agent Interactions Using Log-Contextualized Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2505.17238/) (2025)
- ["My Grade is Wrong!": A Contestable AI Framework for Interactive Feedback in Evaluating Student Essays](https://mlsanity.com/paper/2409.07453/) (2024)
- [CoTAL: Human-in-the-Loop Prompt Engineering for Generalizable Formative Assessment Scoring and Feedback](https://mlsanity.com/paper/2504.02323/) (2025)
- [CoDAE: Adapting Large Language Models for Education via Chain-of-Thought Data Augmentation](https://mlsanity.com/paper/2508.08386/) (2025)
- [Multimodal Methods for Analyzing Learning and Training Environments: A Systematic Literature Review](https://mlsanity.com/paper/2408.14491/) (2024)
- [Towards Multimodal Metaphor Understanding: A Chinese Dataset and Model for Metaphor Mapping Identification](https://mlsanity.com/paper/2501.02434/) (2025)
- [LLMs as Educational Analysts: Transforming Multimodal Data Traces into Actionable Reading Assessment Reports](https://mlsanity.com/paper/2503.02099/) (2025)
- [Evidence-Decision-Feedback: Theory-Driven Adaptive Scaffolding for LLM Agents](https://mlsanity.com/paper/2602.01415/) (2026)
- [Incorporating Metabolic Information into LLMs for Anomaly Detection in Clinical Time-Series](https://mlsanity.com/paper/2410.12830/) (2024)
- [XiHeFusion: Harnessing Large Language Models for Science Communication in Nuclear Fusion](https://mlsanity.com/paper/2502.05615/) (2025)
- [Operationalizing Automated Essay Scoring: A Human-Aware Approach](https://mlsanity.com/paper/2506.21603/) (2025)
- [InqEduAgent: Adaptive AI Learning Partners with Gaussian Process Augmentation](https://mlsanity.com/paper/2508.03174/) (2025)
- [LearnLens: An AI-Enhanced Dashboard to Support Teachers in Open-Ended Classrooms](https://mlsanity.com/paper/2509.10582/) (2025)
- [RubiSCoT: A Framework for AI-Supported Academic Assessment](https://mlsanity.com/paper/2510.17309/) (2025)
- [How Uncertain Is the Grade? A Benchmark of Uncertainty Metrics for LLM-Based Automatic Assessment](https://mlsanity.com/paper/2602.16039/) (2026)
- [AI Misuse in Education Is a Measurement Problem: Toward a Learning Visibility Framework](https://mlsanity.com/paper/2603.07834/) (2026)
- [Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks](https://mlsanity.com/paper/2603.18765/) (2026)
- [Learning to Love Edge Cases in Formative Math Assessment: Using the AMMORE Dataset and Chain-of-Thought Prompting to Improve Grading Accuracy](https://mlsanity.com/paper/2409.17904/) (2024)
- [PEMUTA: Pedagogically-Enriched Multi-Granular Undergraduate Thesis Assessment](https://mlsanity.com/paper/2507.19556/) (2025)
- [Designing Gaze Analytics for ELA Instruction: A User-Centered Dashboard with Conversational AI Support](https://mlsanity.com/paper/2509.03741/) (2025)
- [Bridging Cultural Distance Between Models Default and Local Classroom Demands: How Global Teachers Adopt GenAI to Support Everyday Teaching Practices](https://mlsanity.com/paper/2509.10780/) (2025)
- [Populism Meets AI: Advancing Populism Research with LLMs](https://mlsanity.com/paper/2510.07458/) (2025)

## Abstract (arXiv)

This paper explores the use of large language models (LLMs) to score and explain short-answer assessments in K-12 science. While existing methods can score more structured math and computer science assessments, they often do not provide explanations for the scores. Our study focuses on employing GPT-4 for automated assessment in middle school Earth Science, combining few-shot and active learning with chain-of-thought reasoning. Using a human-in-the-loop approach, we successfully score and provide meaningful explanations for formative assessment responses. A systematic analysis of our method's pros and cons sheds light on the potential for human-in-the-loop techniques to enhance automated grading for open-ended science assessments.