# Decompile-Bench: Million-Scale Binary-Source Function Pairs for Real-World Binary Decompilation

arXiv: https://arxiv.org/abs/2505.12668 · HTML: https://mlsanity.com/paper/2505.12668/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 May 2025.

## Built on by (most cited first, 11 of 11)

- [Evaluating and Improving Automated Repository-Level Rust Issue Resolution with LLM-based Agents](https://mlsanity.com/paper/2602.22764/) (2026)
- [CrackMeBench: Binary Reverse Engineering for Agents](https://mlsanity.com/paper/2605.10597/) (2026)
- [Context-Guided Decompilation: A Step Towards Re-executability](https://mlsanity.com/paper/2511.01763/) (2025)
- [Empowering Autonomous Debugging Agents with Efficient Dynamic Analysis](https://mlsanity.com/paper/2604.24212/) (2026)
- [REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Naming](https://mlsanity.com/paper/2607.07738/) (2026)
- [The Next Challenge for Agentic Cybersecurity: A Realistic, Contamination-Free Reverse Engineering Benchmark](https://mlsanity.com/paper/2608.11469/) (2026)
- [Asm2SrcEval: Evaluating Large Language Models for Assembly-to-Source Code Translation](https://mlsanity.com/paper/2512.00134/) (2025)
- [ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage](https://mlsanity.com/paper/2605.21615/) (2026)
- [When LLM Decompilers Recompile More and Preserve Less](https://mlsanity.com/paper/2609.05370/) (2026)
- [Recompilation Is Not Enough: Test-Guided Decompiled-C Repair](https://mlsanity.com/paper/2609.07201/) (2026)
- [Echo: Learning-based Matching Decompilation using Trusted Back Translation](https://mlsanity.com/paper/2609.18706/) (2026)

## Abstract (arXiv)

Recent advances in LLM-based decompilers have been shown effective to convert low-level binaries into human-readable source code. However, there still lacks a comprehensive benchmark that provides large-scale binary-source function pairs, which is critical for advancing the LLM decompilation technology. Creating accurate binary-source mappings incurs severe issues caused by complex compilation settings and widespread function inlining that obscure the correspondence between binaries and their original source code. Previous efforts have either relied on used contest-style benchmarks, synthetic binary-source mappings that diverge significantly from the mappings in real world, or partially matched binaries with only code lines or variable names, compromising the effectiveness of analyzing the binary functionality. To alleviate these issues, we introduce Decompile-Bench, the first open-source dataset comprising two million binary-source function pairs condensed from 100 million collected function pairs, i.e., 450GB of binaries compiled from permissively licensed GitHub projects. For the evaluation purposes, we also developed a benchmark Decompile-Bench-Eval including manually crafted binaries from the well-established HumanEval and MBPP, alongside the compiled GitHub repositories released after 2025 to mitigate data leakage issues. We further explore commonly-used evaluation metrics to provide a thorough assessment of the studied LLM decompilers and find that fine-tuning with Decompile-Bench causes a 20% improvement over previous benchmarks in terms of the re-executability rate. Our code and data has been released in HuggingFace and Github. https://github.com/albertan017/LLM4Decompile