# From Prompts to Propositions: A Logic-Based Lens on Student-LLM Interactions

arXiv: https://arxiv.org/abs/2504.18691 · HTML: https://mlsanity.com/paper/2504.18691/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 34 references that are in the map.
- First posted on arXiv 25 Apr 2025.

## Builds on (most cited first, 24 of 24)

- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [The Metacognitive Demands and Opportunities of Generative AI](https://mlsanity.com/paper/2312.10893/) (2023)
- [Conversing with Copilot: Exploring Prompt Engineering for Solving CS1 Problems Using Natural Language](https://mlsanity.com/paper/2210.15157/) (2022)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [Comparing Code Explanations Created by Students and Large Language Models](https://mlsanity.com/paper/2304.03938/) (2023)
- [LLM-Assisted Content Analysis: Using Large Language Models to Support Deductive Coding](https://mlsanity.com/paper/2306.14924/) (2023)
- [Computing Education in the Era of Generative AI](https://mlsanity.com/paper/2306.02608/) (2023)
- [The Widening Gap: The Benefits and Harms of Generative AI for Novice Programmers](https://mlsanity.com/paper/2405.17739/) (2024)
- [StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Code](https://mlsanity.com/paper/2306.04556/) (2023)
- [Prompt Problems: A New Programming Exercise for the Generative AI Era](https://mlsanity.com/paper/2311.05943/) (2023)
- [Desirable Characteristics for AI Teaching Assistants in Programming Education](https://mlsanity.com/paper/2405.14178/) (2024)
- [Trust in Generative AI among students: An Exploratory Study](https://mlsanity.com/paper/2310.04631/) (2023)
- [Exploring the Design Space of Cognitive Engagement Techniques with AI-Generated Code for Enhanced Learning](https://mlsanity.com/paper/2410.08922/) (2024)
- [Which is better? Exploring Prompting Strategy For LLM-based Metrics](https://mlsanity.com/paper/2311.03754/) (2023)
- [Student-AI Interaction: A Case Study of CS1 students](https://mlsanity.com/paper/2407.00305/) (2024)
- [Explaining Code with a Purpose: An Integrated Approach for Developing Code Comprehension and Prompting Skills](https://mlsanity.com/paper/2403.06050/) (2024)
- [Towards Detecting Prompt Knowledge Gaps for Improved LLM-guided Issue Resolution](https://mlsanity.com/paper/2501.11709/) (2025)
- [An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issues](https://mlsanity.com/paper/2403.10468/) (2024)
- [Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks](https://mlsanity.com/paper/2410.01026/) (2024)
- [QualiGPT: GPT as an easy-to-use tool for qualitative coding](https://mlsanity.com/paper/2310.07061/) (2023)
- [Substance Beats Style: Why Beginning Students Fail to Code with LLMs](https://mlsanity.com/paper/2410.19792/) (2024)
- [BugSpotter: Automated Generation of Code Debugging Exercises](https://mlsanity.com/paper/2411.14303/) (2024)
- [Revisiting Human Information Foraging: Adaptations for LLM-based Chatbots](https://mlsanity.com/paper/2406.04452/) (2024)

## Built on by (most cited first, 4 of 4)

- [Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI](https://mlsanity.com/paper/2505.19443/) (2025)
- [GenAI Voice Mode in Programming Education](https://mlsanity.com/paper/2509.10596/) (2025)
- [PEPS: Quantum-Inspired Reinforcement Learning for Coherent Reasoning Traces in LLMs](https://mlsanity.com/paper/2509.20105/) (2025)
- [How Much Prompt Is Enough? A Blackbox Minimization of Few-Shots in LLMs](https://mlsanity.com/paper/2609.36289/) (2026)

## Abstract (arXiv)

Background and Context. The increasing integration of large language models (LLMs) in computing education presents an emerging challenge in understanding how students use LLMs and craft prompts to solve computational tasks. Prior research has used both qualitative and quantitative methods to analyze prompting behavior, but these approaches lack scalability or fail to effectively capture the semantic evolution of prompts. Objective. In this paper, we investigate whether students prompts can be systematically analyzed using propositional logic constraints. We examine whether this approach can identify patterns in prompt evolution, detect struggling students, and provide insights into effective and ineffective strategies. Method. We introduce Prompt2Constraints, a novel method that translates students prompts into logical constraints. The constraints are able to represent the intent of the prompts in succinct and quantifiable ways. We used this approach to analyze a dataset of 1,872 prompts from 203 students solving introductory programming tasks. Findings. We find that while successful and unsuccessful attempts tend to use a similar number of constraints overall, when students fail, they often modify their prompts more significantly, shifting problem-solving strategies midway. We also identify points where specific interventions could be most helpful to students for refining their prompts. Implications. This work offers a new and scalable way to detect students who struggle in solving natural language programming tasks. This work could be extended to investigate more complex tasks and integrated into programming tools to provide real-time support.