Trust Me, I Know This Function: Hijacking LLM Static Analysis using Bias
Not on the map. Why?
Abstract (arXiv)
Large Language Models (LLMs) are increasingly trusted to perform automated code review and static analysis at scale, supporting tasks such as vulnerability detection, summarization, and refactoring. In this paper, we identify and exploit a critical vulnerability in LLM-based code analysis: an abstraction bias that causes models to overgeneralize familiar programming patterns and overlook small, meaningful bugs. Adversaries can exploit this blind spot to hijack the control flow of the LLM's interpretation with minimal edits and without affecting actual runtime behavior. We refer to this attack as a Familiar Pattern Attack (FPA). We develop a fully automated, black-box algorithm that discovers and injects FPAs into target code. Our evaluation shows that FPAs are not only effective against basic and reasoning models, but are also transferable across model families (OpenAI, Anthropic, Google), and universal across programming languages (Python, C, Rust, Go). Moreover, FPAs remain effective even when models are explicitly warned about the attack via robust system prompts. Finally, we explore positive, defensive uses of FPAs and discuss their broader implications for the reliability and safety of code-oriented LLMs.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 35 of its 58 references that are in the map. First posted on arXiv 24 Aug 2025.
Builds on
35 of 58 references.
References · 35
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
- 2020Pre-trained Models for Natural Language Processing: A Surveycited by 505
- 2018Generating Natural Language Adversarial Examplescited by 424
- 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 399
- 2018Black-box Generation of Adversarial Text Sequences to Evade Deep Learning Classifierscited by 345
- 2023Are aligned neural networks adversarially aligned?cited by 315
- 2018Adversarial Example Generation with Syntactically Controlled Paraphrase Networkscited by 286
- 2020You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completioncited by 131
- 2018The Adverse Effects of Code Duplication in Machine Learning Models of Codecited by 126
- 2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 93
- 2024Vul-RAG: Enhancing LLM-based Vulnerability Detection via Knowledge-level RAGcited by 68
- 2024Quantifying Contamination in Evaluating Code Generation Capabilities of Language Modelscited by 65
23 more
- 2023TrojanPuzzle: Covertly Poisoning Code-Suggestion Modelscited by 60
- 2023Large Language Models for Code Analysis: Do LLMs Really Do Their Job?cited by 45
- 2024An LLM-Assisted Easy-to-Trigger Backdoor Attack on Code Completion Models: Injecting Disguised Vulnerabilities against Strong Detectioncited by 40
- 2023Unveiling Memorization in Code Modelscited by 30
- 2025Security Concerns for Large Language Models: A Surveycited by 29
- 2024AI-powered Code Review with LLMs: Early Resultscited by 22
- 2023Exploring Code Analysis: Zero-Shot Insights on Syntax and Semantics with LLMscited by 19
- 2024An Empirical Study on the Code Refactoring Capability of Large Language Modelscited by 19
- 2023Poisoned ChatGPT Finds Work for Idle Hands: Exploring Developers' Coding Practices with Insecure Suggestions from Poisoned AI Modelscited by 17
- 2024Outside the Comfort Zone: Analysing LLM Capabilities in Software Vulnerability Detectioncited by 16
- 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 16
- 2024Do Large Code Models Understand Programming Concepts? Counterfactual Analysis for Code Predicatescited by 14
- 2019Generating Black-Box Adversarial Examples for Text Classifiers Using a Deep Reinforced Modelcited by 10
- 2024On Inter-dataset Code Duplication and Data Leakage in Large Language Modelscited by 10
- 2024From Vulnerabilities to Remediation: A Systematic Literature Review of LLMs in Code Securitycited by 10
- 2025Rethinking Code Review Workflows with LLM Assistance: An Empirical Studycited by 10
- 2025SV-TrustEval-C: Evaluating Structure and Semantic Reasoning in Large Language Models for Source Code Vulnerability Analysiscited by 8
- 2025The Code Barrier: What LLMs Actually Understand?cited by 5
- 2025Automated Code Review Using Large Language Models at Ericsson: An Experience Reportcited by 5
- 2024Leveraging Large Language Models for Web Scrapingcited by 3
- 2024Double Backdoored: Converting Code Large Language Model Backdoors to Traditional Malware via Adversarial Instruction Tuning Attackscited by 2
- 2025Web Page Classification using LLMs for Crawling Supportcited by 2
- 2025LLM Contribution Summarization in Software Projectscited by 1
Built on by
Citing papers per year: 2026: 3.
3 papers.
- 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detectioncited by 0
- 2026Predicting Program Exit Code with LLMs and Programming Language Semanticscited by 0
- 2026Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discoverycited by 0
- 2026Cheap to Hypothesize, Costly to Verify: The Defense Surface of Agentic Vulnerability Discovery28 Sep 2026
- 2026Predicting Program Exit Code with LLMs and Programming Language Semantics1 Sep 2026
- 2026Words Speak Louder Than Code: Investigating Cognitive Heuristics in LLM-Based Code Vulnerability Detection29 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2025The Hitchhiker's Guide to Program Analysis, Part II: Deep Thoughts by LLMs
- 2025A Systematic Study of Code Obfuscation Against LLM-based Vulnerability Detection
- 2026Bridging Code Property Graphs and Language Models for Program Analysis ↗
- 2026OpenAnt: LLM-Powered Vulnerability Discovery Through Code Decomposition, Adversarial Verification, and Dynamic Testing
- 2024Can LLM Prompting Serve as a Proxy for Static Analysis in Vulnerability Detection
- 2026An Empirical Evaluation of LLM-Based Approaches for Code Vulnerability Detection: RAG, SFT, and Dual-Agent Systems
- 2026Guiding Symbolic Execution with Static Analysis and LLMs for Vulnerability Discovery
- 2025LLM-Driven Adaptive Source-Sink Identification and False Positive Mitigation for Static Analysis