# Evaluating Neural Decompilation of Dart AOT Binaries: Fine-Tuning, Metric Validity, Specification Leakage, and Reliability

arXiv: https://arxiv.org/abs/2607.06125 · HTML: https://mlsanity.com/paper/2607.06125/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 36 references that are in the map.
- First posted on arXiv 7 Jul 2026.

## Builds on (most cited first, 14 of 14)

- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [Unified Pre-training for Program Understanding and Generation](https://mlsanity.com/paper/2103.06333/) (2021)
- [Unsupervised Translation of Programming Languages](https://mlsanity.com/paper/2006.03511/) (2020)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [How Do Large Language Monkeys Get Their Power (Laws)?](https://mlsanity.com/paper/2502.17578/) (2025)
- [LLM4Decompile: Decompiling Binary Code with Large Language Models](https://mlsanity.com/paper/2403.05286/) (2024)
- [SLaDe: A Portable Small Language Model Decompiler for Optimized Assembly](https://mlsanity.com/paper/2305.12520/) (2023)
- [Refining Decompiled C Code with Large Language Models](https://mlsanity.com/paper/2310.06530/) (2023)
- [SK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin](https://mlsanity.com/paper/2509.22114/) (2025)
- [DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios](https://mlsanity.com/paper/2505.11340/) (2025)
- [Self-Constructed Context Decompilation with Fined-grained Alignment Enhancement](https://mlsanity.com/paper/2406.17233/) (2024)
- [SoK: Potentials and Challenges of Large Language Models for Reverse Engineering](https://mlsanity.com/paper/2509.21821/) (2025)
- [HELIOS: Hierarchical Graph Abstraction for Structure-Aware LLM Decompilation](https://mlsanity.com/paper/2601.14598/) (2026)
- [LLMs as Idiomatic Decompilers: Recovering High-Level Code from x86-64 Assembly for Dart](https://mlsanity.com/paper/2604.02278/) (2026)

## Abstract (arXiv)

We present an execution-based evaluation of neural decompilation for Dart ahead-of-time binaries and an audit of what its scores measure. Across six archived adapter-baseline comparisons, paired tests of pass@k at k = 1, 5, and 10, with Holm adjustment over 18 endpoints, identify functional regressions in both Qwen3-8B adapters at every k. The other four comparisons are inconclusive. On 141 reference-certified, contract-valid tasks, three independently trained graph-prefix systems score the same candidates. Best CodeBLEU has modest association with pass@10 (ρ = .218-.246), compile@10 has weak association (ρ = .072-.082), and only 21.0-23.3% of compiling candidates pass. A paired single-seed intervention that removes semantic names and related cues, while retaining types, arity, and instruction content, reduces coverage from 42/154 to 7/154 tasks. Matched graph perturbations show no detectable degradation under the semantic contract (six-test Holm p >= .750); instruction-use attribution remains unresolved. Across five decoding seeds on MF-174, the baseline solves 4.8 tasks on average, 15 at least once, and one in every seed. We recommend certifying references, aligning metrics on shared candidates, separating metadata from binary input, repeating sampling, and preserving provenance. The released capsule supports integrity checks and replay of archived outcomes.