# When Youth Enter The Chat: An Epistemic Shift in the Validation of LLM-Based Measures of Student Talk

arXiv: https://arxiv.org/abs/2608.23780 · HTML: https://mlsanity.com/paper/2608.23780/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 3 of its 46 references that are in the map.
- First posted on arXiv 24 Aug 2026.

## Builds on (most cited first, 3 of 3)

- [The NCTE Transcripts: A Dataset of Elementary Math Classroom Transcripts](https://mlsanity.com/paper/2211.11772/) (2022)
- ["Don't Forget the Teachers": Towards an Educator-Centered Understanding of Harms from Large Language Models in Education](https://mlsanity.com/paper/2502.14592/) (2025)
- [Optimizing LLM Annotation of Classroom Discourse through Multi-Agent Orchestration](https://mlsanity.com/paper/2603.13353/) (2026)

## Abstract (arXiv)

LLMs are being used increasingly to measure aspects of student discourse (e.g. talk moves, collaboration, equity of voice) at scale. Typically, LLM-based measures of student talk use transcriptions of classroom conversations that only include verbal contributions, which de-contextualize student language. Common practices for validating these measures include comparing outputs against expert annotations by adults, using held out evaluation sets and F1 scores. We argue that these approaches are insufficient to ensure that such measures are meaningful and equitable for teaching and learning, particularly for racially and linguistically marginalized youth. In order to center the youth whose talk is being analyzed, re-contextualizing these classroom conversations and engaging youth in the research process is necessary. Sharing epistemic authority with youth, ultimately, centers their point of view and adds crucial nuance to the analysis of their talk that adult experts, researchers, and LLMs cannot provide. In a case study of multilingual youth in one 8th-grade math classroom, we address the epistemic exclusion of youth by employing multiple ethnographically-oriented methods to re-contextualize student conversations and center youth as epistemic authorities in conversation with researchers and LLMs. We conducted participant observations, interviews, focus groups, and member checks with four focal students. Findings reveal that there were misalignments between students' interpretations of their own math talk experiences and the LLM-based measures of their talk. Students contested both the LLM classifications and the coding scheme used to measure their talk, highlighting the need for youth to be involved in the epistemic process of producing knowledge about their experiences.