# Nova: Generative Language Models for Assembly Code with Hierarchical Attention and Contrastive Learning

arXiv: https://arxiv.org/abs/2311.13721 · HTML: https://mlsanity.com/paper/2311.13721/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 79 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [CodeT: Code Generation with Generated Tests](https://mlsanity.com/paper/2207.10397/) (2022)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [Large Language Models for Code: Security Hardening and Adversarial Testing](https://mlsanity.com/paper/2302.05319/) (2023)
- [Practical Program Repair in the Era of Large Pre-trained Language Models](https://mlsanity.com/paper/2210.14179/) (2022)
- [LongCoder: A Long-Range Pre-trained Language Model for Code Completion](https://mlsanity.com/paper/2306.14893/) (2023)
- [Impact of Code Language Models on Automated Program Repair](https://mlsanity.com/paper/2302.05020/) (2023)
- [Improving Code Generation by Training with Natural Language Feedback](https://mlsanity.com/paper/2303.16749/) (2023)
- [Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2311.12351/) (2023)
- [Neural Network-based Graph Embedding for Cross-Platform Binary Code Similarity Detection](https://mlsanity.com/paper/1708.06525/) (2017)
- [How Effective Are Neural Networks for Fixing Security Vulnerabilities](https://mlsanity.com/paper/2305.18607/) (2023)
- [An Empirical Study of Deep Learning Models for Vulnerability Detection](https://mlsanity.com/paper/2212.08109/) (2022)
- [jTrans: Jump-Aware Transformer for Binary Code Similarity](https://mlsanity.com/paper/2205.12713/) (2022)
- [DIRE: A Neural Approach to Decompiled Identifier Naming](https://mlsanity.com/paper/1909.09029/) (2019)
- [Trex: Learning Execution Semantics from Micro-Traces for Binary Similarity](https://mlsanity.com/paper/2012.08680/) (2020)
- [CAST: Enhancing Code Summarization with Hierarchical Splitting and Reconstruction of Abstract Syntax Trees](https://mlsanity.com/paper/2108.12987/) (2021)
- [SLaDe: A Portable Small Language Model Decompiler for Optimized Assembly](https://mlsanity.com/paper/2305.12520/) (2023)
- [Augmenting Decompiler Output with Learned Variable Names and Types](https://mlsanity.com/paper/2108.06363/) (2021)
- [Beyond the C: Retargetable Decompilation using Neural Machine Translation](https://mlsanity.com/paper/2212.08950/) (2022)
- [Learning to Reverse DNNs from AI Programs Automatically](https://mlsanity.com/paper/2205.10364/) (2022)

## Built on by (most cited first, 12 of 12)

- [LLM4Vuln: A Unified Evaluation Framework for Decoupling and Enhancing LLMs' Vulnerability Reasoning](https://mlsanity.com/paper/2401.16185/) (2024)
- [Skywork UniPic: Unified Autoregressive Modeling for Visual Understanding and Generation](https://mlsanity.com/paper/2508.03320/) (2025)
- [Risk Assessment Framework for Code LLMs via Leveraging Internal States](https://mlsanity.com/paper/2504.14640/) (2025)
- [ShieldedCode: Learning Robust Representations for Virtual Machine Protected Code](https://mlsanity.com/paper/2601.20679/) (2026)
- [HELIOS: Hierarchical Graph Abstraction for Structure-Aware LLM Decompilation](https://mlsanity.com/paper/2601.14598/) (2026)
- [CoDe-R: Refining Decompiler Output with LLMs via Rationale Guidance and Adaptive Inference](https://mlsanity.com/paper/2604.12913/) (2026)
- [A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation](https://mlsanity.com/paper/2602.09112/) (2026)
- [ASSEMBLAGE-DEEPHISTORY: A Cross-Build Binary Dataset with Temporal Coverage](https://mlsanity.com/paper/2605.21615/) (2026)
- [The Periodic Table of LLM Reasoning: A Structured Survey of Reasoning Paradigms, Methods, and Failure Modes](https://mlsanity.com/paper/2606.11470/) (2026)
- [SoK: AI-Augmented Binary Reversing](https://mlsanity.com/paper/2606.17398/) (2026)
- [Instruction Alignment for Binary Code Representation Learning](https://mlsanity.com/paper/2608.11766/) (2026)
- [Recompilation Is Not Enough: Test-Guided Decompiled-C Repair](https://mlsanity.com/paper/2609.07201/) (2026)

## Abstract (arXiv)

Binary code analysis is the foundation of crucial tasks in the security domain; thus building effective binary analysis techniques is more important than ever. Large language models (LLMs) although have brought impressive improvement to source code tasks, do not directly generalize to assembly code due to the unique challenges of assembly: (1) the low information density of assembly and (2) the diverse optimizations in assembly code. To overcome these challenges, this work proposes a hierarchical attention mechanism that builds attention summaries to capture the semantics more effectively and designs contrastive learning objectives to train LLMs to learn assembly optimization. Equipped with these techniques, this work develops Nova, a generative LLM for assembly code. Nova outperforms existing techniques on binary code decompilation by up to 14.84 -- 21.58% (absolute percentage point improvement) higher Pass@1 and Pass@10, and outperforms the latest binary code similarity detection techniques by up to 6.17% Recall@1, showing promising abilities on both assembly generation and understanding tasks.