# Evaluating the Effectiveness of Large Language Models in Solving Simple Programming Tasks: A User-Centered Study

arXiv: https://arxiv.org/abs/2507.04043 · HTML: https://mlsanity.com/paper/2507.04043/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 36 references that are in the map.
- First posted on arXiv 5 Jul 2025.

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Explanations Can Reduce Overreliance on AI Systems During Decision-Making](https://mlsanity.com/paper/2212.06823/) (2022)
- [From LLMs to LLM-based Agents for Software Engineering: A Survey of Current, Challenges and Future](https://mlsanity.com/paper/2408.02479/) (2024)
- [Who Should I Trust: AI or Myself? Leveraging Human and AI Correctness Likelihood to Promote Appropriate Trust in AI-Assisted Decision-Making](https://mlsanity.com/paper/2301.05809/) (2023)
- [Why and When LLM-Based Assistants Can Go Wrong: Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking](https://mlsanity.com/paper/2402.08030/) (2024)
- [AutoP2C: An LLM-Based Agent Framework for Code Repository Generation from Multimodal Content in Academic Papers](https://mlsanity.com/paper/2504.20115/) (2025)
- [A Fine-tuning Dataset and Benchmark for Large Language Models for Protein Understanding](https://mlsanity.com/paper/2406.05540/) (2024)
- ["I'm categorizing LLM as a productivity tool": Examining ethics of LLM use in HCI research practices](https://mlsanity.com/paper/2403.19876/) (2024)
- [AutoM3L: An Automated Multimodal Machine Learning Framework with Large Language Models](https://mlsanity.com/paper/2408.00665/) (2024)
- [Student-AI Interaction: A Case Study of CS1 students](https://mlsanity.com/paper/2407.00305/) (2024)
- [TourSynbio: A Multi-Modal Large Model and Agent Framework to Bridge Text and Protein Sequences for Protein Engineering](https://mlsanity.com/paper/2408.15299/) (2024)
- [DBox: Scaffolding Algorithmic Programming Learning through Learner-LLM Co-Decomposition](https://mlsanity.com/paper/2502.19133/) (2025)
- [ChatCollab: Exploring Collaboration Between Humans and AI Agents in Software Teams](https://mlsanity.com/paper/2412.01992/) (2024)
- [Co-Learning: Code Learning for Multi-Agent Reinforcement Collaborative Framework with Conversational Natural Language Interfaces](https://mlsanity.com/paper/2409.00985/) (2024)
- [Human-AI Experience in Integrated Development Environments: A Systematic Literature Review](https://mlsanity.com/paper/2503.06195/) (2025)
- [TourSynbio-Search: A Large Language Model Driven Agent Framework for Unified Search Method for Protein Engineering](https://mlsanity.com/paper/2411.06024/) (2024)
- [From Automation to Cognition: Redefining the Roles of Educators and Generative AI in Computing Education](https://mlsanity.com/paper/2412.11419/) (2024)
- [ProteinEngine: Empower LLM with Domain Knowledge for Protein Engineering](https://mlsanity.com/paper/2405.06658/) (2024)
- [Reinforcement Fine-Tuning for Reasoning towards Multi-Step Multi-Source Search in Large Language Models](https://mlsanity.com/paper/2506.08352/) (2025)
- [Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants](https://mlsanity.com/paper/2502.07956/) (2025)
- [From Generation to Adaptation: Comparing AI-Assisted Strategies in High School Programming Education](https://mlsanity.com/paper/2506.15955/) (2025)

## Built on by (most cited first, 1 of 1)

- [Exploring Teacher-Chatbot Interaction and Affect in Block-Based Programming](https://mlsanity.com/paper/2603.20211/) (2026)

## Abstract (arXiv)

As large language models (LLMs) become more common in educational tools and programming environments, questions arise about how these systems should interact with users. This study investigates how different interaction styles with ChatGPT-4o (passive, proactive, and collaborative) affect user performance on simple programming tasks. I conducted a within-subjects experiment where fifteen high school students participated, completing three problems under three distinct versions of the model. Each version was designed to represent a specific style of AI support: responding only when asked, offering suggestions automatically, or engaging the user in back-and-forth dialogue.Quantitative analysis revealed that the collaborative interaction style significantly improved task completion time compared to the passive and proactive conditions. Participants also reported higher satisfaction and perceived helpfulness when working with the collaborative version. These findings suggest that the way an LLM communicates, how it guides, prompts, and responds, can meaningfully impact learning and performance. This research highlights the importance of designing LLMs that go beyond functional correctness to support more interactive, adaptive, and user-centered experiences, especially for novice programmers.