mlsanityData to 14 Oct 2026

Cross-Modality Program Representation Learning for Electronic Design Automation with High-Level Synthesis

arXiv 2406.09606 ↗First posted 13 Jun 2024cs.LG cs.AI cs.ARcited by 10

Not on the map. Why?

Abstract (arXiv)

In recent years, domain-specific accelerators (DSAs) have gained popularity for applications such as deep learning and autonomous driving. To facilitate DSA designs, programmers use high-level synthesis (HLS) to compile a high-level description written in C/C++ into a design with low-level hardware description languages that eventually synthesize DSAs on circuits. However, creating a high-quality HLS design still demands significant domain knowledge, particularly in microarchitecture decisions expressed as textitpragmas. Thus, it is desirable to automate such decisions with the help of machine learning for predicting the quality of HLS designs, requiring a deeper understanding of the program that consists of original code and pragmas. Naturally, these programs can be considered as sequence data. In addition, these programs can be compiled and converted into a control data flow graph (CDFG). But existing works either fail to leverage both modalities or combine the two in shallow or coarse ways. We propose ProgSG, a model that allows interaction between the source code sequence modality and the graph modality in a deep and fine-grained way. To alleviate the scarcity of labeled designs, a pre-training method is proposed based on a suite of compiler's data flow analysis tasks. Experimental results show that ProgSG reduces the RMSE of design performance predictions by up to 22%, and identifies designs with an average of 1.10× and 1.26× (up to 8.17× and 13.31×) performance improvement in design space exploration (DSE) task compared to HARP and AutoDSE, respectively.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 43 of its 73 references that are in the map. First posted on arXiv 13 Jun 2024.

Builds on

43 of 73 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  4. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  5. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  6. 2016Variational Graph Auto-Encoderscited by 1,382
  7. 2020CodeBERT: A Pre-Trained Model for Programming and Natural Languagescited by 1,310
  8. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  9. 2018Deep Graph Infomaxcited by 1,106
  10. 2015Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)cited by 1,070
  11. 2021CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generationcited by 906
  12. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 877
31 more
  1. 2020GraphCodeBERT: Pre-training Code Representations with Data Flowcited by 609
  2. 2023Textbooks Are All You Needcited by 569
  3. 2019ERNIE: Enhanced Language Representation with Informative Entitiescited by 528
  4. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  5. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 463
  6. 2017Look, Listen and Learncited by 458
  7. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  8. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  9. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 303
  10. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  11. 2018code2vec: Learning Distributed Representations of Codecited by 250
  12. 2020IntelliCode Compose: Code Generation Using Transformercited by 227
  13. 2022CrossPoint: Self-Supervised Cross-Modal Contrastive Learning for 3D Point Cloud Understandingcited by 198
  14. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  15. 2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
  16. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  17. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  18. 2020Music Gesture for Visual Sound Separationcited by 112
  19. 2019Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labelscited by 111
  20. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  21. 2020GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learningcited by 89
  22. 2023On the Opportunities and Challenges of Foundation Models for Geospatial Artificial Intelligencecited by 88
  23. 2021Machine Learning for Electronic Design Automation: A Surveycited by 72
  24. 2020Self-Supervised Graph Representation Learning via Global Context Predictioncited by 38
  25. 2022Unsupervised Learning for Combinatorial Optimization with Principled Objective Relaxationcited by 32
  26. 2021HANet: Hierarchical Alignment Networks for Video-Text Retrievalcited by 31
  27. 2022High-Level Synthesis Performance Prediction using GNNs: Benchmarking, Modeling, and Advancingcited by 26
  28. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
  29. 2020AutoDSE: Enabling Software Programmers to Design Efficient FPGA Acceleratorscited by 25
  30. 2023Retrieval-based Knowledge Augmented Vision Language Pre-trainingcited by 17
  31. 2024Hardware Phi-1.5B: A Large Language Model Encodes Hardware Domain Specific Knowledgecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025LLM-DSE: Searching Accelerator Parameters with LLM Agentscited by 10
  2. 2025LIFT: LLM-Based Pragma Insertion for HLS via GNN Supervised Fine-Tuningcited by 8
  3. 2025Iceberg: Enhancing HLS Modeling with Synthetic Datacited by 1
  4. 2026MPM-LLM4DSE: Reaching the Pareto Frontier in HLS with Multimodal Learning and LLM-Driven Explorationcited by 1
  5. 2026Report for NSF Workshop on AI for Electronic Design Automationcited by 1
  6. 2026Agent Factories for High Level Synthesis: How Far Can General-Purpose Coding Agents Go in Hardware Optimization?cited by 1
  7. 2025SynCircuit: Automated Generation of New Synthetic RTL Circuits Can Enable Big Data in Circuitscited by 0
  8. 2025Unsupervised Graph Neural Network Framework for Balanced Multipatterning in Advanced Electronic Design Automation Layoutscited by 0
  9. 2026DiffHLS: Differential Learning for High-Level Synthesis QoR Prediction with GNNs and LLM Code Embeddingscited by 0
  10. 2026Agentic Electronic Design Automation: A Handoff Perspectivecited by 0