# "It's not like Jarvis, but it's pretty close!" -- Examining ChatGPT's Usage among Undergraduate Students in Computer Science

arXiv: https://arxiv.org/abs/2311.09651 · HTML: https://mlsanity.com/paper/2311.09651/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 48 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT](https://mlsanity.com/paper/2302.11382/) (2023)
- [Better Zero-Shot Reasoning with Role-Play Prompting](https://mlsanity.com/paper/2308.07702/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [Programming Is Hard -- Or at Least It Used to Be: Educational Opportunities And Challenges of AI Code Generation](https://mlsanity.com/paper/2212.01020/) (2022)
- [Experiences from Using Code Explanations Generated by Large Language Models in a Web Software Development E-Book](https://mlsanity.com/paper/2211.02265/) (2022)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- [Using Large Language Models to Enhance Programming Error Messages](https://mlsanity.com/paper/2210.11630/) (2022)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?](https://mlsanity.com/paper/2303.11146/) (2023)
- [Can GPT-4 Support Analysis of Textual Data in Tasks Requiring Highly Specialized Domain Expertise?](https://mlsanity.com/paper/2306.13906/) (2023)
- [ChatGPT, Can You Generate Solutions for my Coding Exercises? An Evaluation on its Effectiveness in an undergraduate Java Programming Course](https://mlsanity.com/paper/2305.13680/) (2023)
- [Effects of Human vs. Automatic Feedback on Students' Understanding of AI Concepts and Programming Style](https://mlsanity.com/paper/2011.10653/) (2020)

## Built on by (most cited first, 10 of 10)

- [Large Language Models in Computer Science Education: A Systematic Literature Review](https://mlsanity.com/paper/2410.16349/) (2024)
- [Students' Perceptions and Use of Generative AI Tools for Programming Across Different Computing Courses](https://mlsanity.com/paper/2410.06865/) (2024)
- [ChatGPT in Data Visualization Education: A Student Perspective](https://mlsanity.com/paper/2405.00748/) (2024)
- [Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models](https://mlsanity.com/paper/2412.11088/) (2024)
- ["Which LLM should I use?": Evaluating LLMs for tasks performed by Undergraduate Computer Science Students](https://mlsanity.com/paper/2402.01687/) (2024)
- [Finding the Signal in the Noise: An Exploratory Study on Assessing the Effectiveness of AI and Accessibility Forums for Blind Users' Support Needs](https://mlsanity.com/paper/2602.18623/) (2026)
- [BOE-XSUM: Extreme Summarization in Clear Language of Spanish Legal Decrees and Notifications](https://mlsanity.com/paper/2509.24908/) (2025)
- [Compounding Disadvantage: Auditing Intersectional Bias in LLM-Generated Explanations Across Indian and American STEM Education](https://mlsanity.com/paper/2601.14506/) (2026)
- [Patterns of Learner-AI Interaction and Academic Performance in an Object-Oriented Programming Course](https://mlsanity.com/paper/2607.24755/) (2026)
- [Understanding Generative AI Use in Programming MOOCs: The Role of Course Context and Learner Characteristics](https://mlsanity.com/paper/2610.04447/) (2026)

## Abstract (arXiv)

Large language models (LLMs) such as ChatGPT and Google Bard have garnered significant attention in the academic community. Previous research has evaluated these LLMs for various applications such as generating programming exercises and solutions. However, these evaluations have predominantly been conducted by instructors and researchers, not considering the actual usage of LLMs by students. This study adopts a student-first approach to comprehensively understand how undergraduate computer science students utilize ChatGPT, a popular LLM, released by OpenAI. We employ a combination of student surveys and interviews to obtain valuable insights into the benefits, challenges, and suggested improvements related to ChatGPT. Our findings suggest that a majority of students (over 57%) have a convincingly positive outlook towards adopting ChatGPT as an aid in coursework-related tasks. However, our research also highlights various challenges that must be resolved for long-term acceptance of ChatGPT amongst students. The findings from this investigation have broader implications and may be applicable to other LLMs and their role in computing education.