mlsanityData to 7 Oct 2026

Lost at C: A User Study on the Security Implications of Large Language Model Code Assistants

arXiv 2208.09727 ↗First posted 20 Aug 2022cs.CRcited by 87

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) such as OpenAI Codex are increasingly being used as AI-based coding assistants. Understanding the impact of these tools on developers' code is paramount, especially as recent work showed that LLMs may suggest cybersecurity vulnerabilities. We conduct a security-driven user study (N=58) to assess code written by student programmers when assisted by LLMs. Given the potential severity of low-level bugs as well as their relative frequency in real-world projects, we tasked participants with implementing a singly-linked 'shopping list' structure in C. Our results indicate that the security impact in this setting (low-level C with pointer and array manipulations) is small: AI-assisted users produce critical security bugs at a rate no greater than 10% more than the control, indicating the use of LLMs does not introduce new security risks.

Key facts. Cited by 87 papers in the map (data through 7 Oct 2026). First posted on arXiv 20 Aug 2022.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 17; 2024: 24; 2025: 20; 2026: 26.

87 papers.

Sort:

By area:

  1. 2023StarCoder: may the source be with you!cited by 790
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  3. 2023Evaluating Large Language Models: A Comprehensive Surveycited by 200
  4. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 197
  5. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  6. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  7. 2023Who Wrote this Code? Watermarking for Code Generationcited by 116
  8. 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
  9. 2023The Robots are Here: Navigating the Generative AI Revolution in Computing Educationcited by 85
  10. 2023Can ChatGPT replace StackOverflow? A Study on Robustness and Reliability of Large Language Model Code Generationcited by 72
  11. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  12. 2023Clover: Closed-Loop Verifiable Code Generationcited by 62
38 more
  1. 2024Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forwardcited by 44
  2. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  3. 2024Constrained Decoding for Secure Code Generationcited by 35
  4. 2024HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testingcited by 35
  5. 2024Large Language Models in Cybersecurity: State-of-the-Artcited by 34
  6. 2024AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testingcited by 34
  7. 2024Prompting Techniques for Secure Code Generation: A Systematic Investigationcited by 33
  8. 2023StudentEval: A Benchmark of Student-Written Prompts for Large Language Models of Codecited by 32
  9. 2023The FormAI Dataset: Generative AI in Software Security Through the Lens of Formal Verificationcited by 32
  10. 2026Agentic Much? Adoption of Coding Agents on GitHubcited by 31
  11. 2025Speed at the Cost of Quality: How Cursor AI Increases Short-Term Velocity and Long-Term Complexity in Open-Source Projectscited by 23
  12. 2024Depending on yourself when you should: Mentoring LLM with RL agents to become the master in cybersecurity gamescited by 20
  13. 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 18
  14. 2023Semantic Compression With Large Language Modelscited by 17
  15. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  16. 2025Agentic Refactoring: An Empirical Study of AI Coding Agentscited by 17
  17. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 16
  18. 2025"Your AI, My Shell": Demystifying Prompt Injection Attacks on Agentic AI Coding Editorscited by 14
  19. 2024Rocks Coding, Not Development--A Human-Centric, Experimental Evaluation of LLM-Supported SE Taskscited by 13
  20. 2023A Security Risk Taxonomy for Prompt-Based Interaction With Large Language Modelscited by 12
  21. 2024Just another copy and paste? Comparing the security vulnerabilities of ChatGPT generated code and StackOverflow answerscited by 12
  22. 2025A.S.E: A Repository-Level Benchmark for Evaluating Security in AI-Generated Codecited by 12
  23. 2024An Investigation into Misuse of Java Security APIs by Large Language Modelscited by 11
  24. 2024Code Security Vulnerability Repair Using Reinforcement Learning with Large Language Modelscited by 10
  25. 2024Towards More Trustworthy and Interpretable LLMs for Code through Syntax-Grounded Explanationscited by 10
  26. 2025The Invisible Hand: Unveiling Provider Bias in Large Language Models for Code Generationcited by 10
  27. 2025Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferencescited by 10
  28. 2024Guiding Enumerative Program Synthesis with Large Language Modelscited by 9
  29. 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 9
  30. 2023FLAG: Finding Line Anomalies (in code) with Generative AIcited by 7
  31. 2024A Study of Vulnerability Repair in JavaScript Programs with Large Language Modelscited by 7
  32. 2024Enhancing Security of AI-Based Code Synthesis with GitHub Copilot via Cheap and Efficient Prompt-Engineeringcited by 6
  33. 2025Logic Meets Magic: LLMs Cracking Smart Contract Vulnerabilitiescited by 5
  34. 2025Aero-LLM: A Distributed Framework for Secure UAV Communication and Intelligent Decision-Makingcited by 5
  35. 2025Human-AI Experience in Integrated Development Environments: A Systematic Literature Reviewcited by 5
  36. 2025SPENCER: Self-Adaptive Model Distillation for Efficient Code Retrievalcited by 5
  37. 2024When Geoscience Meets Generative AI and Large Language Models: Foundations, Trends, and Future Challengescited by 4
  38. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4

50 most cited of 87 All 87 (JSON)