# Pop Quiz! Can a Large Language Model Help With Reverse Engineering?

arXiv: https://arxiv.org/abs/2202.01142 · HTML: https://mlsanity.com/paper/2202.01142/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 46 references that are in the map.
- First posted on arXiv 2 Feb 2022.

## Builds on (most cited first, 7 of 7)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Text and Code Embeddings by Contrastive Pre-Training](https://mlsanity.com/paper/2201.10005/) (2022)
- [Studying the Usage of Text-To-Text Transfer Transformer to Support Code-Related Tasks](https://mlsanity.com/paper/2102.02017/) (2021)
- [Recommendations for Datasets for Source Code Summarization](https://mlsanity.com/paper/1904.02660/) (2019)
- [Improved Automatic Summarization of Subroutines via Attention to File Context](https://mlsanity.com/paper/2004.04881/) (2020)
- [XDA: Accurate, Robust Disassembly with Transfer Learning](https://mlsanity.com/paper/2010.00770/) (2020)
- [ICSREF: A Framework for Automated Reverse Engineering of Industrial Control Systems Binaries](https://mlsanity.com/paper/1812.03478/) (2018)

## Built on by (most cited first, 14 of 14)

- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [Automatic Generation of Programming Exercises and Code Explanations using Large Language Models](https://mlsanity.com/paper/2206.11861/) (2022)
- [When LLMs Meet Cybersecurity: A Systematic Literature Review](https://mlsanity.com/paper/2405.03644/) (2024)
- [Large Language Models for Cyber Security: A Systematic Literature Review](https://mlsanity.com/paper/2405.04760/) (2024)
- ["It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers](https://mlsanity.com/paper/2304.02491/) (2023)
- [InstructExcel: A Benchmark for Natural Language Instruction in Excel](https://mlsanity.com/paper/2310.14495/) (2023)
- [Codex Hacks HackerRank: Memorization Issues and a Framework for Code Synthesis Evaluation](https://mlsanity.com/paper/2212.02684/) (2022)
- [Exploring the Efficacy of Large Language Models (GPT-4) in Binary Reverse Engineering](https://mlsanity.com/paper/2406.06637/) (2024)
- [Aero-LLM: A Distributed Framework for Secure UAV Communication and Intelligent Decision-Making](https://mlsanity.com/paper/2502.05220/) (2025)
- [Automatically Generating Rules of Malicious Software Packages via Large Language Model](https://mlsanity.com/paper/2504.17198/) (2025)
- [StackSight: Unveiling WebAssembly through Large Language Models and Neurosymbolic Chain-of-Thought Decompilation](https://mlsanity.com/paper/2406.04568/) (2024)
- [LLMs as Firmware Experts: A Runtime-Grown Tree-of-Agents Framework](https://mlsanity.com/paper/2511.18438/) (2025)
- [Leveraging LLMs for Grammar Adaptation: A Study on Metamodel-Grammar Co-Evolution](https://mlsanity.com/paper/2605.21465/) (2026)
- [Humans vs. Machines: A Comparison With AI Agents in Software Reverse Engineering](https://mlsanity.com/paper/2610.14960/) (2026)

## Abstract (arXiv)

Large language models (such as OpenAI's Codex) have demonstrated impressive zero-shot multi-task capabilities in the software domain, including code explanation. In this work, we examine if this ability can be used to help with reverse engineering. Specifically, we investigate prompting Codex to identify the purpose, capabilities, and important variable names or values from code, even when the code is produced through decompilation. Alongside an examination of the model's responses in answering open-ended questions, we devise a true/false quiz framework to characterize the performance of the language model. We present an extensive quantitative analysis of the measured performance of the language model on a set of program purpose identification and information extraction tasks: of the 136,260 questions we posed, it answered 72,754 correctly. A key takeaway is that while promising, LLMs are not yet ready for zero-shot reverse engineering.