# SEVRA-BENCH: Social Engineering of Vulnerabilities in Review Agents

arXiv: https://arxiv.org/abs/2606.13757 · HTML: https://mlsanity.com/paper/2606.13757/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 47 of its 83 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 30 of 47)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [SWE-bench: Can Language Models Resolve Real-World GitHub Issues?](https://mlsanity.com/paper/2310.06770/) (2023)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs](https://mlsanity.com/paper/2401.06373/) (2024)
- [Identifying the Risks of LM Agents with an LM-Emulated Sandbox](https://mlsanity.com/paper/2309.15817/) (2023)
- [Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributions](https://mlsanity.com/paper/2108.09293/) (2021)
- [AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Bases](https://mlsanity.com/paper/2407.12784/) (2024)
- [Formalizing and Benchmarking Prompt Injection Attacks and Defenses](https://mlsanity.com/paper/2310.12815/) (2023)
- [Do Users Write More Insecure Code with AI Assistants?](https://mlsanity.com/paper/2211.03622/) (2022)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion](https://mlsanity.com/paper/2007.02220/) (2020)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [ReCode: Robustness Evaluation of Code Generation Models](https://mlsanity.com/paper/2212.10264/) (2022)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [CVE-Bench: A Benchmark for AI Agents' Ability to Exploit Real-World Web Application Vulnerabilities](https://mlsanity.com/paper/2503.17332/) (2025)
- [CVEfixes: Automated Collection of Vulnerabilities and Their Fixes from Open-Source Software](https://mlsanity.com/paper/2107.08760/) (2021)
- [CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale](https://mlsanity.com/paper/2506.02548/) (2025)
- [Backstabber's Knife Collection: A Review of Open Source Software Supply Chain Attacks](https://mlsanity.com/paper/2005.09535/) (2020)
- [TrojanPuzzle: Covertly Poisoning Code-Suggestion Models](https://mlsanity.com/paper/2301.02344/) (2023)
- [CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generation](https://mlsanity.com/paper/2501.08200/) (2025)
- [Taxonomy of Attacks on Open-Source Software Supply Chains](https://mlsanity.com/paper/2204.04008/) (2022)
- [BaxBench: Can LLMs Generate Correct and Secure Backends?](https://mlsanity.com/paper/2502.11844/) (2025)
- [SALLM: Security Assessment of Generated Code](https://mlsanity.com/paper/2311.00889/) (2023)
- [Small World with High Risks: A Study of Security Threats in the npm Ecosystem](https://mlsanity.com/paper/1902.09217/) (2019)
- [CodeAgent: Autonomous Communicative Agents for Code Review](https://mlsanity.com/paper/2402.02172/) (2024)
- [Multi-target Backdoor Attacks for Code Pre-trained Models](https://mlsanity.com/paper/2306.08350/) (2023)
- [LLM Can be a Dangerous Persuader: Empirical Study of Persuasion Safety in Large Language Models](https://mlsanity.com/paper/2504.10430/) (2025)
- [AI-powered Code Review with LLMs: Early Results](https://mlsanity.com/paper/2404.18496/) (2024)
- [BitsAI-CR: Automated Code Review via LLM in Practice](https://mlsanity.com/paper/2501.15134/) (2025)

## Built on by (most cited first, 1 of 1)

- ["You're Right, Let Me Fix It": How LLM Agents Damage Correct Work When Falsely Accused](https://mlsanity.com/paper/2609.32616/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are increasingly deployed in automated code-review systems, where their approvals can determine which code is merged into shared repositories. However, it is unclear whether review agents can detect vulnerability-introducing code when an attacker controls both the code change and the persuasive Pull Request (PR) narrative designed to mask it. We introduce SEVRA-BENCH (Social Engineering of Vulnerabilities in Review Agents), a benchmark that measures how often a review agent approves such adversarial PR s. Each PR in SEVRA-BENCH is built from a historical commit that fixed a vulnerability. We automatically reverse that fix to extract the original vulnerable code, and submit the resulting code change as a PR wrapped in one of 15 social-engineering framings. To test review-agent resilience to narrative manipulation, these framings vary dimensions such as supporting evidence, conveyed urgency, signals of prior approval, and appeals to authority. SEVRA-BENCH evaluates a retained challenge split of roughly 1000 adversarial PRs drawn from publicly disclosed vulnerability fixes across the top 10 entries of the MITRE's 2025 most dangerous software weaknesses. Evaluating 8 review agents against this benchmark, we reveal that review agents are susceptible to narrative manipulation, exposing a significant gap in security capabilities.