# A Meta-Analysis of LLM Effects on Students across Qualification, Socialisation, and Subjectification

arXiv: https://arxiv.org/abs/2509.22725 · HTML: https://mlsanity.com/paper/2509.22725/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 138 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Builds on (most cited first, 15 of 15)

- [Improving Language Model Negotiation with Self-Play and In-Context Learning from AI Feedback](https://mlsanity.com/paper/2305.10142/) (2023)
- [Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Survey](https://mlsanity.com/paper/2305.18703/) (2023)
- [On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?](https://mlsanity.com/paper/2303.11146/) (2023)
- [Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues](https://mlsanity.com/paper/2503.06424/) (2025)
- [GPTutor: a ChatGPT-powered programming tool for code explanation](https://mlsanity.com/paper/2305.01863/) (2023)
- [A Transparency Index Framework for AI in Education](https://mlsanity.com/paper/2206.03220/) (2022)
- [Generative Artificial Intelligence in Higher Education: Evidence from an Analysis of Institutional Policies and Guidelines](https://mlsanity.com/paper/2402.01659/) (2024)
- [GPT-4 as a Homework Tutor can Improve Student Engagement and Learning Outcomes](https://mlsanity.com/paper/2409.15981/) (2024)
- [Learning-by-teaching with ChatGPT: The effect of teachable ChatGPT agent on programming education](https://mlsanity.com/paper/2412.15226/) (2024)
- [DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition](https://mlsanity.com/paper/2502.19133/) (2025)
- [More AI Assistance Reduces Cognitive Engagement: Examining the AI Assistance Dilemma in AI-Supported Note-Taking](https://mlsanity.com/paper/2509.03392/) (2025)
- [Students' Reliance on AI in Higher Education: Identifying Contributing Factors](https://mlsanity.com/paper/2506.13845/) (2025)
- [Generative AI and Agency in Education: A Critical Scoping Review and Thematic Analysis](https://mlsanity.com/paper/2411.00631/) (2024)
- [How Students Use AI Feedback Matters: Experimental Evidence on Physics Achievement and Autonomy](https://mlsanity.com/paper/2505.08672/) (2025)
- [COLP: Scaffolding Children's Online Long-term Collaborative Learning](https://mlsanity.com/paper/2502.03226/) (2025)

## Built on by (most cited first, 2 of 2)

- [How unique are hallucinated citations offered by generative Artificial Intelligence models?](https://mlsanity.com/paper/2604.16407/) (2026)
- [Evaluating Epistemic Guardrails in AI Reading Assistants: A Behavioral Audit of a Minimal Prototype](https://mlsanity.com/paper/2604.27275/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly positioned as solutions for education, yet evaluations often reduce their impact to narrow performance metrics. This paper reframes the question by asking "what kind of impact should LLMs have in education?" Drawing on Biesta's tripartite account of good education: qualification, socialisation, and subjectification, we present a meta-analysis of 133 experimental and quasi-experimental studies (k = 188). Overall, the impact of LLMs on student learning is positive but uneven. Strong effects emerge in qualification, particularly when LLMs function as tutors in sustained interventions. Socialisation outcomes appear more variable, concentrated in sustained, reflective interventions. Subjectification, linked to autonomy and learner development, remains fragile, with improvements confined to small-scale, long-term studies. This purpose-level view highlights design as the decisive factor: without scaffolds for participation and agency, LLMs privilege what is easiest to measure while neglecting broader aims of education. For HCI and education, the issue is not just whether LLMs work, but what futures they enable or foreclose.