# Humans vs. Machines: A Comparison With AI Agents in Software Reverse Engineering

arXiv: https://arxiv.org/abs/2610.14960 · HTML: https://mlsanity.com/paper/2610.14960/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 49 references that are in the map.
- First posted on arXiv 10 Oct 2026.

## Placed in

- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 17 of 17)

- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Vulnerability Detection with Code Language Models: How Far Are We?](https://mlsanity.com/paper/2403.18624/) (2024)
- [IRIS: LLM-Assisted Static Analysis for Detecting Security Vulnerabilities](https://mlsanity.com/paper/2405.17238/) (2024)
- [LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks](https://mlsanity.com/paper/2312.12575/) (2023)
- [MCP-Universe: Benchmarking Large Language Models with Real-World Model Context Protocol Servers](https://mlsanity.com/paper/2508.14704/) (2025)
- [LLMs in Software Security: A Survey of Vulnerability Detection Techniques and Insights](https://mlsanity.com/paper/2502.07049/) (2025)
- [Large Language Models for Code Analysis: Do LLMs Really Do Their Job?](https://mlsanity.com/paper/2310.12357/) (2023)
- [An Observational Investigation of Reverse Engineers' Processes](https://mlsanity.com/paper/1912.00317/) (2019)
- [Measuring and Augmenting Large Language Models for Solving Capture-the-Flag Challenges](https://mlsanity.com/paper/2506.17644/) (2025)
- [Pop Quiz! Can a Large Language Model Help With Reverse Engineering?](https://mlsanity.com/paper/2202.01142/) (2022)
- [Decoding the Secrets of Machine Learning in Malware Classification: A Deep Dive into Datasets, Feature Extraction, and Model Performance](https://mlsanity.com/paper/2307.14657/) (2023)
- [Towards Effective Offensive Security LLM Agents: Hyperparameter Tuning, LLM as a Judge, and a Lightweight CTF Benchmark](https://mlsanity.com/paper/2508.05674/) (2025)
- [Exploring the Efficacy of Large Language Models (GPT-4) in Binary Reverse Engineering](https://mlsanity.com/paper/2406.06637/) (2024)
- [ReCopilot: Reverse Engineering Copilot in Binary Analysis](https://mlsanity.com/paper/2505.16366/) (2025)
- [Deconstructing Obfuscation: A four-dimensional framework for evaluating Large Language Models assembly code deobfuscation capabilities](https://mlsanity.com/paper/2505.19887/) (2025)
- [An Empirical Study on the Effectiveness of Large Language Models for Binary Code Understanding](https://mlsanity.com/paper/2504.21803/) (2025)
- [Challenges and Future Directions in Agentic Reverse Engineering Systems](https://mlsanity.com/paper/2604.14317/) (2026)

## Abstract (arXiv)

Can an autonomous AI agent reverse engineer a binary as well as a skilled human analyst? Do they adopt the same strategy, and can agents scale to real-world programs? We compare human and LLM-based software reverse engineering (SRE): nine models tackle two compact static-analysis challenges previously solved by human analysts, scored with a common understanding rubric, and we record their complete interactions with IDA Pro. This lets us compare not only what humans and agents recover, but also how they investigate a binary. The best agents (GPT-5.5 Pro, Claude Opus 4.8, and Kimi K3) achieve statistically human-competitive understanding on both tasks, while weaker models remain far behind. Ablations show that the gap depends on code representation and workflow demands, not on model capability alone: open-weight models improve substantially with richer code representations, whereas frontier models show no significant change. Similar success rates hide different workflows. Humans repeatedly revisit functions and follow local program structure; strong agents jump across the binary, retrieve information broadly, and rarely return to inspected code, a behavior we call greedy fetching. This strategy is not fixed: given semantically richer code, agents fetch less without losing understanding. Because greedy fetching aggregates large amounts of code in the agent's context, we measure context-window saturation across 4,000 real-world binaries: large windows accommodate many programs, but larger applications and lower-level representations quickly exhaust them. Autonomous reverse engineering is becoming competitive, with a large speed advantage, but scaling it will require better navigation, selective retrieval, and persistent analysis state, not merely larger context windows.