# DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios

arXiv: https://arxiv.org/abs/2505.11340 · HTML: https://mlsanity.com/paper/2505.11340/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 40 references that are in the map.
- First posted on arXiv 16 May 2025.

## Builds on (most cited first, 12 of 12)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [MoDeGPT: Modular Decomposition for Large Language Model Compression](https://mlsanity.com/paper/2408.09632/) (2024)
- [LLM4Decompile: Decompiling Binary Code with Large Language Models](https://mlsanity.com/paper/2403.05286/) (2024)
- [KIEval: A Knowledge-grounded Interactive Evaluation Framework for Large Language Models](https://mlsanity.com/paper/2402.15043/) (2024)
- [Evaluating Hallucinations in Chinese Large Language Models](https://mlsanity.com/paper/2310.03368/) (2023)
- [Refining Decompiled C Code with Large Language Models](https://mlsanity.com/paper/2310.06530/) (2023)
- [Aligning Large Language Models by On-Policy Self-Judgment](https://mlsanity.com/paper/2402.11253/) (2024)
- [Beyond the C: Retargetable Decompilation using Neural Machine Translation](https://mlsanity.com/paper/2212.08950/) (2022)
- [DeepCRCEval: Revisiting the Evaluation of Code Review Comment Generation](https://mlsanity.com/paper/2412.18291/) (2024)
- [Comparative Analysis of Listwise Reranking with Large Language Models in Limited-Resource Language Contexts](https://mlsanity.com/paper/2412.20061/) (2024)

## Built on by (most cited first, 4 of 4)

- [CrackMeBench: Binary Reverse Engineering for Agents](https://mlsanity.com/paper/2605.10597/) (2026)
- [Context-Guided Decompilation: A Step Towards Re-executability](https://mlsanity.com/paper/2511.01763/) (2025)
- [PCodeTrans: Translate Decompiled Pseudocode to Compilable and Executable Equivalent](https://mlsanity.com/paper/2603.14855/) (2026)
- [Evaluating Neural Decompilation of Dart AOT Binaries: Fine-Tuning, Metric Validity, Specification Leakage, and Reliability](https://mlsanity.com/paper/2607.06125/) (2026)

## Abstract (arXiv)

Decompilers are fundamental tools for critical security tasks, from vulnerability discovery to malware analysis, yet their evaluation remains fragmented. Existing approaches primarily focus on syntactic correctness through synthetic micro-benchmarks or subjective human ratings, failing to address real-world requirements for semantic fidelity and analyst usability. We present DecompileBench, the first comprehensive framework that enables effective evaluation of decompilers in reverse engineering workflows through three key components: textitreal-world function extraction (comprising 23,400 functions from 130 real-world programs), textitruntime-aware validation, and textitautomated human-centric assessment using LLM-as-Judge to quantify the effectiveness of decompilers in reverse engineering workflows. Through a systematic comparison between six industrial-strength decompilers and six recent LLM-powered approaches, we demonstrate that LLM-based methods surpass commercial tools in code understandability despite 52.2% lower functionality correctness. These findings highlight the potential of LLM-based approaches to transform human-centric reverse engineering. We open source hrefhttps://github.com/Jennieett/DecompileBenchDecompileBench to provide a framework to advance research on decompilers and assist security experts in making informed tool selections based on their specific requirements.