mlsanityData to 7 Oct 2026

Binary Code Summarization: Benchmarking ChatGPT/GPT-4 and Other Large Language Models

arXiv 2312.09601 ↗First posted 15 Dec 2023cs.CR cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Binary code summarization, while invaluable for understanding code semantics, is challenging due to its labor-intensive nature. This study delves into the potential of large language models (LLMs) for binary code comprehension. To this end, we present BinSum, a comprehensive benchmark and dataset of over 557K binary functions and introduce a novel method for prompt synthesis and optimization. To more accurately gauge LLM performance, we also propose a new semantic similarity metric that surpasses traditional exact-match approaches. Our extensive evaluation of prominent LLMs, including ChatGPT, GPT-4, Llama 2, and Code Llama, reveals 10 pivotal insights. This evaluation generates 4 billion inference tokens, incurred a total expense of 11,418 US dollars and 873 NVIDIA A100 GPU hours. Our findings highlight both the transformative potential of LLMs in this field and the challenges yet to be overcome.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 33 of its 79 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

33 of 79 references.

References · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022Emergent Abilities of Large Language Modelscited by 2,483
  7. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  10. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,286
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  12. 2023Large Language Models as Optimizerscited by 830
21 more
  1. 2022Large Language Models Can Self-Improvecited by 656
  2. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 638
  3. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  4. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 600
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  6. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  7. 2020Deep Learning Based Text Classification: A Comprehensive Reviewcited by 140
  8. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  9. 2023Guiding Large Language Models via Directional Stimulus Promptingcited by 94
  10. 2021PalmTree: Learning an Assembly Language Model for Instruction Embeddingcited by 49
  11. 2021On the Evaluation of Neural Code Summarizationcited by 47
  12. 2022jTrans: Jump-Aware Transformer for Binary Code Similaritycited by 44
  13. 2016A Survey of Symbolic Execution Techniquescited by 36
  14. 2020Trex: Learning Execution Semantics from Micro-Traces for Binary Similaritycited by 34
  15. 2023Extending Source Code Pre-Trained Language Models to Summarise Decompiled Binariescited by 25
  16. 2021Augmenting Decompiler Output with Learned Variable Names and Typescited by 23
  17. 2020XDA: Accurate, Robust Disassembly with Transfer Learningcited by 21
  18. 2019An Observational Investigation of Reverse Engineers' Processescited by 20
  19. 2021Variable Name Recovery in Decompiled Binary Code using Constrained Masked Language Modelingcited by 7
  20. 2021XFL: Naming Functions in Binaries with Extreme Multi-label Learningcited by 5
  21. 2022NeuDep: Neural Binary Memory Dependence Analysiscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 4; 2026: 2.

13 papers.

Sort:

  1. 2024Large Language Models for Forecasting and Anomaly Detection: A Systematic Literature Reviewcited by 50
  2. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  3. 2024Divide-and-Conquer Meets Consensus: Unleashing the Power of Functions in Code Generationcited by 19
  4. 2025Large Language Models (LLMs) for Source Code Analysis: applications, models and datasetscited by 15
  5. 2024Source Code Foundation Models are Transferable Binary Analysis Knowledge Basescited by 8
  6. 2024Enhancing Source Code Security with LLMs: Demystifying The Challenges and Generating Reliable Repairscited by 4
  7. 2025ASMA-Tune: Unlocking LLMs' Assembly Code Comprehension via Structural-Semantic Instruction Tuningcited by 4
  8. 2023Graphene: Infrastructure Security Posture Analysis with AI-generated Attack Graphscited by 3
  9. 2025Benchmarking AI Models in Software Engineering: A Review, Search Tool, and Unified Approach for Elevating Benchmark Qualitycited by 3
  10. 2024Unmasking the Shadows: Pinpoint the Implementations of Anti-Dynamic Analysis Techniques in Malware Using LLMcited by 1
  11. 2026REFORGE: A Method for Benchmarking LLMs' Reverse Engineering Capabilities in Decompiled Binary Function Namingcited by 1
  12. 2025Retrofit: Continual Learning with Controlled Forgetting for Binary Security Detection and Analysiscited by 0
1 more
  1. 2026Large Byte Model: Teaching Language Models About Compiled Codecited by 0