Not Everyone Wins with LLMs: Behavioral Patterns and Pedagogical Implications for AI Literacy in Programmatic Data Science
Not on the map. Why?
Abstract (arXiv)
LLMs promise to democratize technical work in complex domains like programmatic data analysis, but not everyone benefits equally. We study how students with varied experiences use LLMs to complete Python-based data analysis in computational notebooks in a graduate course. Drawing on homework logs, recordings, and surveys from 36 students, we ask: Which experience matters most, and how does it shape AI use? Our mixed-methods analysis shows that technical experience -- not AI familiarity or communication skills -- remains a significant predictor of success. Students also vary widely in how they leverage LLMs, struggling at stages of forming intent, expressing inputs, interpreting outputs, and assessing results. We identify success and failure behaviors, such as providing context or decomposing prompts, that distinguish effective use. These findings inform AI literacy interventions, highlighting that lightweight demonstrations improve surface fluency but are insufficient; deeper training and scaffolds are needed to cultivate resilient AI use skills.
Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Sep 2025.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2026: 6.
6 papers.
- 2026Reflections and New Directions for Human-Centered Large Language Modelscited by 2
- 2026RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactionscited by 1
- 2026Bespoke Visual Assistance: What and How do Blind and Low-Vision People Create with Agentic Programming?cited by 0
- 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysiscited by 0
- 2026TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Educationcited by 0
- 2026DataCanvas-EDU: An Agentic Framework for Instructor-Guided Synthetic Data Generation in Business Analytics Educationcited by 0
- 2026DataCanvas-EDU: An Agentic Framework for Instructor-Guided Synthetic Data Generation in Business Analytics Education17 Sep 2026
- 2026TutorTrace: A Dataset and Taxonomy for Classifying Learner Behavioral States during AI-Assisted Programming Education22 Aug 2026
- 2026AdaLens: Interactive Storyline for Monitoring and Steering Long-Running Agentic Data Analysis18 Aug 2026
- 2026Bespoke Visual Assistance: What and How do Blind and Low-Vision People Create with Agentic Programming?23 Jul 2026
- 2026Reflections and New Directions for Human-Centered Large Language Models7 May 2026
- 2026RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions1 May 2026