mlsanityData to 7 Oct 2026

Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning

arXiv 2311.13721 ↗First posted 22 Nov 2023cs.SE cs.AIcited by 12

Not on the map. Why?

Abstract (arXiv)

Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques is more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations in assembly code. To overcome these challenges, this work proposes a hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively and designs contrastive learning objectives to train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova, a generative LLM for assembly code. Nova outperforms existing techniques on binary code decompilation by up to 14.84 -- 21.58% (absolute percentage point improvement) higher Pass@1 and Pass@10, and outperforms the latest binary code similarity detection techniques by up to 6.17% Recall@1, showing promising abilities on both assembly generation and understanding tasks.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 32 of its 79 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

32 of 79 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 890
  11. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  12. 2022CodeT: Code Generation with Generated Testscited by 369
20 more
  1. 2022CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learningcited by 342
  2. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  3. 2022Practical Program Repair in the Era of Large Pre-trained Language Modelscited by 133
  4. 2023LongCoder: A Long-Range Pre-trained Language Model for Code Completioncited by 110
  5. 2023Impact of Code Language Models on Automated Program Repaircited by 85
  6. 2023Improving Code Generation by Training with Natural Language Feedbackcited by 75
  7. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 71
  8. 2017Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detectioncited by 65
  9. 2023How Effective Are Neural Networks for Fixing Security Vulnerabilitiescited by 50
  10. 2022jTrans: Jump-Aware Transformer for Binary Code Similaritycited by 44
  11. 2022An Empirical Study of Deep Learning Models for Vulnerability Detectioncited by 44
  12. 2019DIRE: A Neural Approach to Decompiled Identifier Namingcited by 36
  13. 2020Trex: Learning Execution Semantics from Micro-Traces for Binary Similaritycited by 34
  14. 2021CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Treescited by 31
  15. 2023SLaDe: A Portable Small Language Model Decompiler for Optimized Assemblycited by 28
  16. 2021Augmenting Decompiler Output with Learned Variable Names and Typescited by 23
  17. 2022Beyond the C: Retargetable Decompilation using Neural Machine Translationcited by 15
  18. 2022Learning to Reverse DNNs from AI Programs Automaticallycited by 11
  19. 2022HierarchyNet: Learning to Summarize Source Code with Heterogeneous Representationscited by 9
  20. 2022Decompiling x86 Deep Neural Network Executablescited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 9.

12 papers.

  1. 2024LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoningcited by 57
  2. 2025Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generationcited by 16
  3. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 8
  4. 2026ShieldedCode: Learning Robust Representations for Virtual Machine Protected Codecited by 8
  5. 2026HELIOS: Hierarchical Graph Abstraction for Structure-Aware LLM Decompilationcited by 3
  6. 2026CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inferencecited by 1
  7. 2026A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentationcited by 0
  8. 2026ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coveragecited by 0
  9. 2026The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modescited by 0
  10. 2026SoK: AI-Augmented Binary Reversingcited by 0
  11. 2026Instruction Alignment for Binary Code Representation Learningcited by 0
  12. 2026Recompilation Is Not Enough: Test-Guided Decompiled-C Repaircited by 0