# Cross-Modality Program Representation Learning for Electronic Design Automation with High-Level Synthesis arXiv: https://arxiv.org/abs/2406.09606 · HTML: https://mlsanity.com/paper/2406.09606/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 43 of its 73 references that are in the map. - First posted on arXiv 13 Jun 2024. ## Builds on (most cited first, 30 of 43) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019) - [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023) - [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023) - [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021) - [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016) - [CodeBERT: A Pre-Trained Model for Programming and Natural Languages](https://mlsanity.com/paper/2002.08155/) (2020) - [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020) - [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018) - [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015) - [CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation](https://mlsanity.com/paper/2109.00859/) (2021) - [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018) - [GraphCodeBERT: Pre-training Code Representations with Data Flow](https://mlsanity.com/paper/2009.08366/) (2020) - [Textbooks Are All You Need](https://mlsanity.com/paper/2306.11644/) (2023) - [ERNIE: Enhanced Language Representation with Informative Entities](https://mlsanity.com/paper/1905.07129/) (2019) - [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020) - [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020) - [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017) - [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019) - [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022) - [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020) - [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023) - [code2vec: Learning Distributed Representations of Code](https://mlsanity.com/paper/1803.09473/) (2018) - [IntelliCode Compose: Code Generation Using Transformer](https://mlsanity.com/paper/2005.08025/) (2020) - [CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understanding](https://mlsanity.com/paper/2203.00680/) (2022) - [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023) - [Self-Supervised Learning of Graph Neural Networks: A Unified Review](https://mlsanity.com/paper/2102.10757/) (2021) - [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023) - [Deep Bidirectional Language-Knowledge Graph Pretraining](https://mlsanity.com/paper/2210.09338/) (2022) - [Music Gesture for Visual Sound Separation](https://mlsanity.com/paper/2004.09476/) (2020) ## Built on by (most cited first, 10 of 10) - [LLM-DSE: Searching Accelerator Parameters with LLM Agents](https://mlsanity.com/paper/2505.12188/) (2025) - [LIFT: LLM-Based Pragma Insertion for HLS via GNN Supervised Fine-Tuning](https://mlsanity.com/paper/2504.21187/) (2025) - [Iceberg: Enhancing HLS Modeling with Synthetic Data](https://mlsanity.com/paper/2507.09948/) (2025) - [MPM-LLM4DSE: Reaching the Pareto Frontier in HLS with Multimodal Learning and LLM-Driven Exploration](https://mlsanity.com/paper/2601.04801/) (2026) - [Report for NSF Workshop on AI for Electronic Design Automation](https://mlsanity.com/paper/2601.14541/) (2026) - [Agent Factories for High Level Synthesis: How Far Can General-Purpose Coding Agents Go in Hardware Optimization?](https://mlsanity.com/paper/2603.25719/) (2026) - [SynCircuit: Automated Generation of New Synthetic RTL Circuits Can Enable Big Data in Circuits](https://mlsanity.com/paper/2509.00071/) (2025) - [Unsupervised Graph Neural Network Framework for Balanced Multipatterning in Advanced Electronic Design Automation Layouts](https://mlsanity.com/paper/2511.16374/) (2025) - [DiffHLS: Differential Learning for High-Level Synthesis QoR Prediction with GNNs and LLM Code Embeddings](https://mlsanity.com/paper/2604.09240/) (2026) - [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026) ## Abstract (arXiv) In recent years, domain-specific accelerators (DSAs) have gained popularity for applications such as deep learning and autonomous driving. To facilitate DSA designs, programmers use high-level synthesis (HLS) to compile a high-level description written in C/C++ into a design with low-level hardware description languages that eventually synthesize DSAs on circuits. However, creating a high-quality HLS design still demands significant domain knowledge, particularly in microarchitecture decisions expressed as textitpragmas. Thus, it is desirable to automate such decisions with the help of machine learning for predicting the quality of HLS designs, requiring a deeper understanding of the program that consists of original code and pragmas. Naturally, these programs can be considered as sequence data. In addition, these programs can be compiled and converted into a control data flow graph (CDFG). But existing works either fail to leverage both modalities or combine the two in shallow or coarse ways. We propose ProgSG, a model that allows interaction between the source code sequence modality and the graph modality in a deep and fine-grained way. To alleviate the scarcity of labeled designs, a pre-training method is proposed based on a suite of compiler's data flow analysis tasks. Experimental results show that ProgSG reduces the RMSE of design performance predictions by up to 22%, and identifies designs with an average of 1.10× and 1.26× (up to 8.17× and 13.31×) performance improvement in design space exploration (DSE) task compared to HARP and AutoDSE, respectively.