# Learning Agent-based Modeling with LLM Companions: Experiences of Novices and Experts Using ChatGPT & NetLogo Chat

arXiv: https://arxiv.org/abs/2401.17163 · HTML: https://mlsanity.com/paper/2401.17163/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 116 references that are in the map.
- First posted on arXiv 30 Jan 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [The Impact of AI on Developer Productivity: Evidence from GitHub Copilot](https://mlsanity.com/paper/2302.06590/) (2023)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [GitHub Copilot AI pair programmer: Asset or Liability?](https://mlsanity.com/paper/2206.15331/) (2022)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Low-resource Languages: A Review of Past Work and Future Challenges](https://mlsanity.com/paper/2006.07264/) (2020)
- [Studying the effect of AI Code Generators on Supporting Novice Learners in Introductory Programming](https://mlsanity.com/paper/2302.07427/) (2023)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [Repair Is Nearly Generation: Multilingual Program Repair with LLMs](https://mlsanity.com/paper/2208.11640/) (2022)
- [Grammar Prompting for Domain-Specific Language Generation with Large Language Models](https://mlsanity.com/paper/2305.19234/) (2023)
- [What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents](https://mlsanity.com/paper/1901.06525/) (2019)
- [Thrilled by Your Progress! Large Language Models (GPT-4) No Longer Struggle to Pass Assessments in Higher Education Programming Courses](https://mlsanity.com/paper/2306.10073/) (2023)
- [Automatic Code Documentation Generation Using GPT-3](https://mlsanity.com/paper/2209.02235/) (2022)
- [On the Transferability of Pre-trained Language Models for Low-Resource Programming Languages](https://mlsanity.com/paper/2204.09653/) (2022)
- [On the Design of AI-powered Code Assistants for Notebooks](https://mlsanity.com/paper/2301.11178/) (2023)
- [Beyond Generating Code: Evaluating GPT on a Data Visualization Course](https://mlsanity.com/paper/2306.02914/) (2023)
- [The potential of LLMs for coding with low-resource and domain-specific programming languages](https://mlsanity.com/paper/2307.13018/) (2023)
- [MyCrunchGPT: A chatGPT assisted framework for scientific machine learning](https://mlsanity.com/paper/2306.15551/) (2023)
- [MultiCoder: Multi-Programming-Lingual Pre-Training for Low-Resource Code Completion](https://mlsanity.com/paper/2212.09666/) (2022)

## Built on by (most cited first, 23 of 23)

- [Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward](https://mlsanity.com/paper/2406.07765/) (2024)
- [MathVC: An LLM-Simulated Multi-Character Virtual Classroom for Mathematics Education](https://mlsanity.com/paper/2404.06711/) (2024)
- [What Should We Engineer in Prompts? Training Humans in Requirement-Driven LLM Use](https://mlsanity.com/paper/2409.08775/) (2024)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Large Language Model-Enabled Multi-Agent Manufacturing Systems](https://mlsanity.com/paper/2406.01893/) (2024)
- [Beyond the Hype: A Comprehensive Review of Current Trends in Generative AI Research, Teaching Practices, and Tools](https://mlsanity.com/paper/2412.14732/) (2024)
- [Can Large Language Models Identify Implicit Suicidal Ideation? An Empirical Evaluation](https://mlsanity.com/paper/2502.17899/) (2025)
- [Broadening Access to Simulations for End-Users via Large Language Models: Challenges and Opportunities](https://mlsanity.com/paper/2409.15290/) (2024)
- [Large Language Models for Agent-Based Modelling: Current and possible uses across the modelling cycle](https://mlsanity.com/paper/2507.05723/) (2025)
- [Carbon and Silicon, Coexist or Compete? A Survey on Human-AI Interactions in Agent-based Modeling and Simulation](https://mlsanity.com/paper/2502.18145/) (2025)
- [Novice Developers' Perspectives on Adopting LLMs for Software Development: A Systematic Literature Review](https://mlsanity.com/paper/2503.07556/) (2025)
- [From Consumption to Collaboration: Measuring Interaction Patterns to Augment Human Cognition in Open-Ended Tasks](https://mlsanity.com/paper/2504.02780/) (2025)
- ["Would You Want an AI Tutor?" Understanding Stakeholder Perceptions of LLM-based Systems in the Classroom](https://mlsanity.com/paper/2503.02885/) (2025)
- [Using (Not-so) Large Language Models to Generate Simulation Models in a Formal DSL: A Study on Reaction Networks](https://mlsanity.com/paper/2503.01675/) (2025)
- [Unveiling the Learning Mind of Language Models: A Cognitive Framework and Empirical Study](https://mlsanity.com/paper/2506.13464/) (2025)
- [Is Passive Expertise-Based Personalization Enough? A Case Study in AI-Assisted Test-Taking](https://mlsanity.com/paper/2511.23376/) (2025)
- [False Friends in the Shell: Unveiling the Emoticon Semantic Confusion in Large Language Models](https://mlsanity.com/paper/2601.07885/) (2026)
- [On Integrating Resilience and Human Oversight into LLM-Assisted Modeling Workflows for Digital Twins](https://mlsanity.com/paper/2603.25898/) (2026)
- [Priming, Path-dependence, and Plasticity: Understanding the molding of user-LLM interaction and its implications from (many) chat logs in the wild](https://mlsanity.com/paper/2605.05767/) (2026)
- [Feasibility of AI-Assisted Programming for End-User Development](https://mlsanity.com/paper/2512.05666/) (2025)
- [Upskilling with Generative AI: Practices and Challenges for Freelance Knowledge Workers](https://mlsanity.com/paper/2604.27231/) (2026)
- [Artificial Intelligence and Modeling & Simulation: An Overview](https://mlsanity.com/paper/2608.00366/) (2026)
- [Clarify User Expertise: Towards Proactive Conversational Agents Tailoring Responses to User Proficiency](https://mlsanity.com/paper/2608.22266/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have the potential to fundamentally change the way people engage in computer programming. Agent-based modeling (ABM) has become ubiquitous in natural and social sciences and education, yet no prior studies have explored the potential of LLMs to assist it. We designed NetLogo Chat to support the learning and practice of NetLogo, a programming language for ABM. To understand how users perceive, use, and need LLM-based interfaces, we interviewed 30 participants from global academia, industry, and graduate schools. Experts reported more perceived benefits than novices and were more inclined to adopt LLMs in their workflow. We found significant differences between experts and novices in their perceptions, behaviors, and needs for human-AI collaboration. We surfaced a knowledge gap between experts and novices as a possible reason for the benefit gap. We identified guidance, personalization, and integration as major needs for LLM-based interfaces to support the programming of ABM.