# Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning

arXiv: https://arxiv.org/abs/2410.08922 · HTML: https://mlsanity.com/paper/2410.08922/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 125 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 25 of 25)

- [To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making](https://mlsanity.com/paper/2102.09692/) (2021)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [The Metacognitive Demands and Opportunities of Generative AI](https://mlsanity.com/paper/2312.10893/) (2023)
- [The Robots are Here: Navigating the Generative AI Revolution in Computing Education](https://mlsanity.com/paper/2310.00658/) (2023)
- [CodeAid: Evaluating a Classroom Deployment of an LLM-based Programming Assistant that Balances Student and Educator Needs](https://mlsanity.com/paper/2401.11314/) (2024)
- [CodeHelp: Using Large Language Models with Guardrails for Scalable Support in Programming Classes](https://mlsanity.com/paper/2308.06921/) (2023)
- [How Novices Use LLM-Based Code Generators to Solve CS1 Coding Tasks in a Self-Paced Learning Environment](https://mlsanity.com/paper/2309.14049/) (2023)
- [Do People Engage Cognitively with AI? Impact of AI Assistance on Incidental Learning](https://mlsanity.com/paper/2202.05402/) (2022)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Prompt Problems: A New Programming Exercise for the Generative AI Era](https://mlsanity.com/paper/2311.05943/) (2023)
- [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024)
- [How Do Analysts Understand and Verify AI-Assisted Data Analyses?](https://mlsanity.com/paper/2309.10947/) (2023)
- [How Do Data Analysts Respond to AI Assistance? A Wizard-of-Oz Study](https://mlsanity.com/paper/2309.10108/) (2023)
- [The Effects of Generative AI on Computing Students' Help-Seeking Preferences](https://mlsanity.com/paper/2401.02262/) (2024)
- [Generative AI in Computing Education: Perspectives of Students and Instructors](https://mlsanity.com/paper/2308.04309/) (2023)
- [Ivie: Lightweight Anchored Explanations of Just-Generated Code](https://mlsanity.com/paper/2403.02491/) (2024)
- [More Robots are Coming: Large Multimodal Models (ChatGPT) can Solve Visually Diverse Images of Parsons Problems](https://mlsanity.com/paper/2311.04926/) (2023)
- [CodeTailor: LLM-Powered Personalized Parsons Puzzles for Engaging Support While Learning Programming](https://mlsanity.com/paper/2401.12125/) (2024)
- [Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills](https://mlsanity.com/paper/2403.06050/) (2024)
- [How Novice Programmers Use and Experience ChatGPT when Solving Programming Exercises in an Introductory Course](https://mlsanity.com/paper/2407.20792/) (2024)
- [Analyzing Chat Protocols of Novice Programmers Solving Introductory Programming Tasks with ChatGPT](https://mlsanity.com/paper/2405.19132/) (2024)
- [TraceDiff: Debugging Unexpected Code Behavior Using Trace Divergences](https://mlsanity.com/paper/1708.03786/) (2017)
- [Novices' Learning Barriers When Using Code Examples in Open-Ended Programming](https://mlsanity.com/paper/2104.11806/) (2021)

## Built on by (most cited first, 16 of 16)

- [LLMs' Reshaping of People, Processes, Products, and Society in Software Development: A Comprehensive Exploration with Early Adopters](https://mlsanity.com/paper/2503.05012/) (2025)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [CorpusStudio: Surfacing Emergent Patterns in a Corpus of Prior Work while Writing](https://mlsanity.com/paper/2503.12436/) (2025)
- [From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions](https://mlsanity.com/paper/2504.18691/) (2025)
- [Semantic Commit: Helping Users Update Intent Specifications for AI Memory at Scale](https://mlsanity.com/paper/2504.09283/) (2025)
- [StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagation](https://mlsanity.com/paper/2508.03182/) (2025)
- [Coding With AI: From a Reflection on Industrial Practices to Future Computer Science and Software Engineering Education](https://mlsanity.com/paper/2512.23982/) (2025)
- [From Answer Givers to Design Mentors: Guiding LLMs with the Cognitive Apprenticeship Model](https://mlsanity.com/paper/2601.19053/) (2026)
- [Beyond Autocomplete: Designing CopilotLens Towards Transparent and Explainable AI Coding Agents](https://mlsanity.com/paper/2506.20062/) (2025)
- [Single Conversation Methodology: A Human-Centered Protocol for AI-Assisted Software Development](https://mlsanity.com/paper/2507.12665/) (2025)
- [Impact of Multimodal and Conversational AI on Learning Outcomes and Experience](https://mlsanity.com/paper/2604.02221/) (2026)
- [Agents That Teach: Towards Designing Incidental Learning Back into AI-Assisted Software Development](https://mlsanity.com/paper/2607.06101/) (2026)
- [From Execution to Education: A Bloom-Aligned Framework for Measuring Educational Control in LLMs](https://mlsanity.com/paper/2607.08009/) (2026)
- [Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course](https://mlsanity.com/paper/2607.24755/) (2026)
- [AgentForge: An Immersive Role-Playing Platform for Learning Agentic Software Engineering](https://mlsanity.com/paper/2608.04148/) (2026)
- [VibeJam: A User Study Platform for Web Development with Agents](https://mlsanity.com/paper/2608.29889/) (2026)

## Abstract (arXiv)

Novice programmers are increasingly relying on Large Language Models (LLMs) to generate code for learning programming concepts. However, this interaction can lead to superficial engagement, giving learners an illusion of learning and hindering skill development. To address this issue, we conducted a systematic design exploration to develop seven cognitive engagement techniques aimed at promoting deeper engagement with AI-generated code. In this paper, we describe our design process, the initial seven techniques and results from a between-subjects study (N=82). We then iteratively refined the top techniques and further evaluated them through a within-subjects study (N=42). We evaluate the friction each technique introduces, their effectiveness in helping learners apply concepts to isomorphic tasks without AI assistance, and their success in aligning learners' perceived and actual coding abilities. Ultimately, our results highlight the most effective technique: guiding learners through the step-by-step problem-solving process, where they engage in an interactive dialog with the AI, prompting what needs to be done at each stage before the corresponding code is revealed.