# Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction

arXiv: https://arxiv.org/abs/2306.03090 · HTML: https://mlsanity.com/paper/2306.03090/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 42 references that are in the map.
- First posted on arXiv 5 Jun 2023.

## Builds on (most cited first, 13 of 13)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators](https://mlsanity.com/paper/2303.16854/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification](https://mlsanity.com/paper/2303.03953/) (2023)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [Learning gain differences between ChatGPT and human tutor generated algebra hints](https://mlsanity.com/paper/2302.06871/) (2023)
- [Using Transformers to Provide Teachers with Personalized Feedback on their Classroom Discourse: The TalkMoves Application](https://mlsanity.com/paper/2105.07949/) (2021)
- [Better by you, better than me, chatgpt3 as writing assistance in students essays](https://mlsanity.com/paper/2302.04536/) (2023)

## Built on by (most cited first, 30 of 37)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMs](https://mlsanity.com/paper/2402.03927/) (2024)
- [Bridging the Novice-Expert Gap via Models of Decision-Making: A Case Study on Remediating Math Mistakes](https://mlsanity.com/paper/2310.10648/) (2023)
- [Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise](https://mlsanity.com/paper/2410.03017/) (2024)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Teach AI How to Code: Using Large Language Models as Teachable Agents for Programming Education](https://mlsanity.com/paper/2309.14534/) (2023)
- [LLM-as-a-tutor in EFL Writing Education: Focusing on Evaluation of Student-LLM Interaction](https://mlsanity.com/paper/2310.05191/) (2023)
- [Formal Theorem Proving by Rewarding LLMs to Decompose Proofs Hierarchically](https://mlsanity.com/paper/2411.01829/) (2024)
- [A Material Lens on Coloniality in NLP](https://mlsanity.com/paper/2311.08391/) (2023)
- [Flesch or Fumble? Evaluating Readability Standard Alignment of Instruction-Tuned Language Models](https://mlsanity.com/paper/2309.05454/) (2023)
- [A Comprehensive Survey on Evaluating Large Language Model Applications in the Medical Industry](https://mlsanity.com/paper/2404.15777/) (2024)
- [Adapting Large Language Models for Education: Foundational Capabilities, Potentials, and Challenges](https://mlsanity.com/paper/2401.08664/) (2023)
- [Evaluating and Optimizing Educational Content with Large Language Model Judgments](https://mlsanity.com/paper/2403.02795/) (2024)
- [Edu-ConvoKit: An Open-Source Library for Education Conversation Data](https://mlsanity.com/paper/2402.05111/) (2024)
- [Improving Assessment of Tutoring Practices using Retrieval-Augmented Generation](https://mlsanity.com/paper/2402.14594/) (2024)
- [Using Large Language Models to Assess Tutors' Performance in Reacting to Students Making Math Errors](https://mlsanity.com/paper/2401.03238/) (2024)
- [The Promises and Pitfalls of Using Language Models to Measure Instruction Quality in Education](https://mlsanity.com/paper/2404.02444/) (2024)
- [The StudyChat Dataset: Analyzing Student Dialogues With ChatGPT in an Artificial Intelligence Course](https://mlsanity.com/paper/2503.07928/) (2025)
- [Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback](https://mlsanity.com/paper/2310.01132/) (2023)
- [A Survey on Human-AI Collaboration with Large Foundation Models](https://mlsanity.com/paper/2403.04931/) (2024)
- [Automated Assessment of Encouragement and Warmth in Classrooms Leveraging Multimodal Emotional Features and ChatGPT](https://mlsanity.com/paper/2404.15310/) (2024)
- [LLMs in Education: Novel Perspectives, Challenges, and Opportunities](https://mlsanity.com/paper/2409.11917/) (2024)
- [Opportunities and Challenges of LLMs in Education: An NLP Perspective](https://mlsanity.com/paper/2507.22753/) (2025)
- [When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challenges](https://mlsanity.com/paper/2402.03349/) (2024)
- [Analyzing Large Language Models for Classroom Discussion Assessment](https://mlsanity.com/paper/2406.08680/) (2024)
- ["Mistakes Help Us Grow": Facilitating and Evaluating Growth Mindset Supportive Language in Classrooms](https://mlsanity.com/paper/2310.10637/) (2023)
- [Multimodal Assessment of Classroom Discourse Quality: A Text-Centered Attention-Based Multi-Task Learning Approach](https://mlsanity.com/paper/2505.07902/) (2025)
- [Knowledge without Wisdom: Measuring Misalignment between LLMs and Intended Impact](https://mlsanity.com/paper/2603.00883/) (2026)
- [Exploring the effectiveness of ChatGPT-based feedback compared with teacher feedback and self-feedback: Evidence from Chinese to English translation](https://mlsanity.com/paper/2309.01645/) (2023)
- [Exploring the Capabilities of ChatGPT in Ancient Chinese Translation and Person Name Recognition](https://mlsanity.com/paper/2312.15304/) (2023)

## Abstract (arXiv)

Coaching, which involves classroom observation and expert feedback, is a widespread and fundamental part of teacher training. However, the majority of teachers do not have access to consistent, high quality coaching due to limited resources and access to expertise. We explore whether generative AI could become a cost-effective complement to expert feedback by serving as an automated teacher coach. In doing so, we propose three teacher coaching tasks for generative AI: (A) scoring transcript segments based on classroom observation instruments, (B) identifying highlights and missed opportunities for good instructional strategies, and (C) providing actionable suggestions for eliciting more student reasoning. We recruit expert math teachers to evaluate the zero-shot performance of ChatGPT on each of these tasks for elementary math classroom transcripts. Our results reveal that ChatGPT generates responses that are relevant to improving instruction, but they are often not novel or insightful. For example, 82% of the model's suggestions point to places in the transcript where the teacher is already implementing that suggestion. Our work highlights the challenges of producing insightful, novel and truthful feedback for teachers while paving the way for future research to address these obstacles and improve the capacity of generative AI to coach teachers.