# Beyond Text Matching: Towards Reference-Free Evaluation for Human-Oriented Binary Reverse Engineering

arXiv: https://arxiv.org/abs/2608.07038 · HTML: https://mlsanity.com/paper/2608.07038/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 28 of its 69 references that are in the map.
- First posted on arXiv 7 Aug 2026.

## Builds on (most cited first, 28 of 28)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Qwen2.5-Coder Technical Report](https://mlsanity.com/paper/2409.12186/) (2024)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance](https://mlsanity.com/paper/1909.02622/) (2019)
- [DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence](https://mlsanity.com/paper/2406.11931/) (2024)
- [CodeBERTScore: Evaluating Code Generation with Pretrained Models of Code](https://mlsanity.com/paper/2302.05527/) (2023)
- [Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineering](https://mlsanity.com/paper/2502.06193/) (2025)
- [CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences](https://mlsanity.com/paper/2403.09032/) (2024)
- [ICE-Score: Instructing Large Language Models to Evaluate Code](https://mlsanity.com/paper/2304.14317/) (2023)
- [Does BLEU Score Work for Code Migration?](https://mlsanity.com/paper/1906.04903/) (2019)
- [Can LLMs Replace Manual Annotation of Software Engineering Artifacts?](https://mlsanity.com/paper/2408.05534/) (2024)
- [Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binaries](https://mlsanity.com/paper/2301.01701/) (2023)
- [Refining Decompiled C Code with Large Language Models](https://mlsanity.com/paper/2310.06530/) (2023)
- [Augmenting Decompiler Output with Learned Variable Names and Types](https://mlsanity.com/paper/2108.06363/) (2021)
- [Leveraging LLMs for the Quality Assurance of Software Requirements](https://mlsanity.com/paper/2408.10886/) (2024)
- [Source Code Foundation Models are Transferable Binary Analysis Knowledge Bases](https://mlsanity.com/paper/2405.19581/) (2024)
- [LLMs as Evaluators: A Novel Approach to Evaluate Bug Report Summarization](https://mlsanity.com/paper/2409.00630/) (2024)
- [MALSIGHT: Exploring Malicious Source Code and Benign Pseudocode for Iterative Binary Malware Summarization](https://mlsanity.com/paper/2406.18379/) (2024)
- [ReCopilot: Reverse Engineering Copilot in Binary Analysis](https://mlsanity.com/paper/2505.16366/) (2025)
- [A Novel HDL Code Generator for Effectively Testing FPGA Logic Synthesis Compilers](https://mlsanity.com/paper/2407.12037/) (2024)
- [FoC: Figure out the Cryptographic Functions in Stripped Binaries with LLMs](https://mlsanity.com/paper/2403.18403/) (2024)
- [Evaluating Generated Commit Messages with Large Language Models](https://mlsanity.com/paper/2507.10906/) (2025)
- [FidelityGPT: Correcting Decompilation Distortions with Retrieval Augmented Generation](https://mlsanity.com/paper/2510.19615/) (2025)

## Abstract (arXiv)

Human-Oriented Binary Reverse Engineering (HOBRE) aims to transform decompiled pseudocode into a more human-friendly representation, thereby reducing the cognitive burden of reverse analysis and improving efficiency. However, reliably evaluating HOBRE outputs remains a fundamental challenge: human evaluation is costly, time-consuming, and difficult to scale, while existing automated metrics either require executable test cases and runtime environments that are often unavailable for real-world binaries, or rely on high-quality source code references that are typically inaccessible and fail to capture semantically equivalent but lexically diverse outputs. Although LLM-as-a-Judge paradigm is naturally well-suited to HOBRE evaluation, its effectiveness remains underexplored. This paper presents the first systematic investigation of the LLM-as-a-Judge paradigm for HOBRE across three representative tasks: function name recovery, binary code summarization, and decompilation optimization. We introduce BinJudgeBench, the first expert-annotated, reference-free evaluation benchmark based on multi-dimensional human judgment, where LLM-as-a-Judge achieves an average correlation of 63.20% with human judgment, outperforming traditional automated metrics at 35.04%. By analyzing judge configurations across backbone LLMs, prompting strategies, and decoding temperatures, we find that no ``one-size-fits-all'' configuration exists, as the optimal setup varies across tasks and individual samples. To address this, we propose BinJudge, which employs a lightweight routing mechanism to adaptively select the optimal judge configuration for each task and sample. BinJudge improves correlation with human experts by 4.5%-24.7% and reduces API cost to 0.06×-0.84× of that of static best configurations, providing a scalable, cost-effective, and high-fidelity automated evaluation scheme for HOBRE.