# EduBehaviors: Assertion-based Schemas for Auditable Coding of Educational Dialogues

arXiv: https://arxiv.org/abs/2609.27043 · HTML: https://mlsanity.com/paper/2609.27043/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 27 of its 33 references that are in the map.
- First posted on arXiv 22 Sep 2026.

## Builds on (most cited first, 27 of 27)

- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [Decomposed Prompting: A Modular Approach for Solving Complex Tasks](https://mlsanity.com/paper/2210.02406/) (2022)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Can Large Language Models Transform Computational Social Science?](https://mlsanity.com/paper/2305.03514/) (2023)
- [FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets](https://mlsanity.com/paper/2307.10928/) (2023)
- [Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences](https://mlsanity.com/paper/2404.12272/) (2024)
- [Efficient Few-Shot Learning Without Prompts](https://mlsanity.com/paper/2209.11055/) (2022)
- [Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooM](https://mlsanity.com/paper/2404.12259/) (2024)
- [Automated Annotation with Generative AI Requires Validation](https://mlsanity.com/paper/2306.00176/) (2023)
- [Testing the Reliability of ChatGPT for Text Annotation and Classification: A Cautionary Remark](https://mlsanity.com/paper/2304.11085/) (2023)
- [Tutor CoPilot: A Human-AI Approach for Scaling Real-Time Expertise](https://mlsanity.com/paper/2410.03017/) (2024)
- [Measuring Conversational Uptake: A Case Study on Student-Teacher Interactions](https://mlsanity.com/paper/2106.03873/) (2021)
- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- [The TalkMoves Dataset: K-12 Mathematics Lesson Transcripts Annotated for Teacher and Student Discursive Moves](https://mlsanity.com/paper/2204.09652/) (2022)
- [Codebook LLMs: Evaluating LLMs as Measurement Tools for Political Science Concepts](https://mlsanity.com/paper/2407.10747/) (2024)
- [Interpretable-by-Design Text Understanding with Iteratively Generated Concept Bottleneck](https://mlsanity.com/paper/2310.19660/) (2023)
- [Evaluating Large Language Models in Analysing Classroom Dialogue](https://mlsanity.com/paper/2402.02380/) (2024)
- [Prompting in the Dark: Assessing Human Performance in Prompt Engineering for Data Labeling When Gold Labels Are Absent](https://mlsanity.com/paper/2502.11267/) (2025)
- [Edu-ConvoKit: An Open-Source Library for Education Conversation Data](https://mlsanity.com/paper/2402.05111/) (2024)
- [Augmenting Human-Annotated Training Data with Large Language Model Generation and Distillation in Open-Response Assessment](https://mlsanity.com/paper/2501.09126/) (2025)
- [Leveraging LLMs to Assess Tutor Moves in Real-Life Dialogues: A Feasibility Study](https://mlsanity.com/paper/2506.17410/) (2025)
- [The Measurement Revolution? Credible Measurement and Inference in the Age of AI](https://mlsanity.com/paper/2608.23524/) (2026)
- [How well do Large Language Models Recognize Instructional Moves? Establishing Baselines for Foundation Models in Educational Discourse](https://mlsanity.com/paper/2512.19903/) (2025)
- [LLM Reasoning Predicts When Models Are Right: Evidence from Coding Classroom Discourse](https://mlsanity.com/paper/2602.09832/) (2026)
- [Sandpiper: Orchestrated AI-Annotation for Educational Discourse at Scale](https://mlsanity.com/paper/2603.08406/) (2026)

## Abstract (arXiv)

Large language models have enabled rapid deployment of language classification and analysis, allowing users to describe complex constructs in natural language, such as pedagogical moves or expressions of learning and engagement, and label them in conversational datasets such as classroom or tutoring transcripts. However, because LLM reasoning is opaque, we have no verifiable, mechanistic insight into why a model chose a label for an utterance, and output differences over time or across models challenge replication. We introduce the EduBehaviors framework to generate interpretable, reliable, and scalable construct annotations in educational dialogue data that retain human agency while leveraging the strengths of generative AI, encoders, and machine learning models. We evaluate the framework on the TalkMoves dataset, predicting the Teacher TalkMoves labels. Our best configuration results in a macro-F1 of 0.764 and 0.756 Cohen's kappa, demonstrating improvements over direct prompting approaches and showing competitiveness with fine-tuned classifiers while gaining the benefit of greater transparency and comparability. To facilitate adoption and operationalization of the EduBehaviors framework in educational dialogue data by other researchers, we also release the EduBehaviors Toolkit with two complementary open-source tools based on the TalkMoves application.