# ChipNeMo: Domain-Adapted LLMs for Chip Design

arXiv: https://arxiv.org/abs/2311.00176 · HTML: https://mlsanity.com/paper/2311.00176/

## Key facts

- Cited by 174 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 69 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 30 of 48)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Multitask Prompted Training Enables Zero-Shot Task Generalization](https://mlsanity.com/paper/2110.08207/) (2021)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [Atlas: Few-shot Learning with Retrieval Augmented Language Models](https://mlsanity.com/paper/2208.03299/) (2022)
- [BloombergGPT: A Large Language Model for Finance](https://mlsanity.com/paper/2303.17564/) (2023)
- [CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis](https://mlsanity.com/paper/2203.13474/) (2022)
- [RACE: Large-scale ReAding Comprehension Dataset From Examinations](https://mlsanity.com/paper/1704.04683/) (2017)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)

## Built on by (most cited first, 30 of 174)

- [Tower: An Open Multilingual Large Language Model for Translation-Related Tasks](https://mlsanity.com/paper/2402.17733/) (2024)
- [BetterV: Controlled Verilog Generation with Discriminative Guidance](https://mlsanity.com/paper/2402.03375/) (2024)
- [AnalogCoder: Analog Circuit Design via Training-Free Code Generation](https://mlsanity.com/paper/2405.14918/) (2024)
- [MG-Verilog: Multi-grained Dataset Towards Enhanced LLM-assisted Verilog Generation](https://mlsanity.com/paper/2407.01910/) (2024)
- [WirelessLLM: Empowering Large Language Models Towards Wireless Intelligence](https://mlsanity.com/paper/2405.17053/) (2024)
- [ADO-LLM: Analog Design Bayesian Optimization with In-Context Learning of Large Language Models](https://mlsanity.com/paper/2406.18770/) (2024)
- [Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTS](https://mlsanity.com/paper/2402.03289/) (2024)
- [A Survey of Research in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2501.09655/) (2025)
- [LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automation](https://mlsanity.com/paper/2401.12224/) (2023)
- [CodeV: Empowering LLMs with HDL Generation through Multi-Level Summarization](https://mlsanity.com/paper/2407.10424/) (2024)
- [AnalogGenie: A Generative Engine for Automatic Discovery of Analog Circuit Topologies](https://mlsanity.com/paper/2503.00205/) (2025)
- [Data is all you need: Finetuning LLMs for Chip Design via an Automated design-data augmentation framework](https://mlsanity.com/paper/2403.11202/) (2024)
- [AutoVCoder: A Systematic Framework for Automated Verilog Code Generation using LLMs](https://mlsanity.com/paper/2407.18333/) (2024)
- [MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning](https://mlsanity.com/paper/2405.12130/) (2024)
- [RTLRewriter: Methodologies for Large Models aided RTL Code Optimization](https://mlsanity.com/paper/2409.11414/) (2024)
- [MAGE: A Multi-Agent Engine for Automated RTL Code Generation](https://mlsanity.com/paper/2412.07822/) (2024)
- [Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA](https://mlsanity.com/paper/2407.15353/) (2024)
- [HDLdebugger: Streamlining HDL debugging with Large Language Models](https://mlsanity.com/paper/2403.11671/) (2024)
- [RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projects](https://mlsanity.com/paper/2405.17378/) (2024)
- [Large Language Models for Manufacturing](https://mlsanity.com/paper/2410.21418/) (2024)
- [DeepRTL: Bridging Verilog Understanding and Generation with a Unified Representation Model](https://mlsanity.com/paper/2502.15832/) (2025)
- [LLMs and the Future of Chip Design: Unveiling Security Risks and Building Trust](https://mlsanity.com/paper/2405.07061/) (2024)
- [OmniScience: A Domain-Specialized LLM for Scientific Reasoning and Discovery](https://mlsanity.com/paper/2503.17604/) (2025)
- [CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation](https://mlsanity.com/paper/2404.08806/) (2024)
- [Automatically Improving LLM-based Verilog Generation using EDA Tool Feedback](https://mlsanity.com/paper/2411.11856/) (2024)
- [PyraNet: A Multi-Layered Hierarchical Dataset for Verilog](https://mlsanity.com/paper/2412.06947/) (2024)
- [AnalogXpert: Automating Analog Topology Synthesis by Incorporating Circuit Design Expertise into Large Language Models](https://mlsanity.com/paper/2412.19824/) (2024)
- [RTL++: Graph-enhanced LLM for RTL Code Generation](https://mlsanity.com/paper/2505.13479/) (2025)
- [HDLCoRe: A Training-Free Framework for Mitigating Hallucinations in LLM-Generated HDL](https://mlsanity.com/paper/2503.16528/) (2025)
- [SymRTLO: Enhancing RTL Code Optimization with LLMs and Neuron-Inspired Symbolic Reasoning](https://mlsanity.com/paper/2504.10369/) (2025)

## Abstract (arXiv)

ChipNeMo aims to explore the applications of large language models (LLMs) for industrial chip design. Instead of directly deploying off-the-shelf commercial or open-source LLMs, we instead adopt the following domain adaptation techniques: domain-adaptive tokenization, domain-adaptive continued pretraining, model alignment with domain-specific instructions, and domain-adapted retrieval models. We evaluate these methods on three selected LLM applications for chip design: an engineering assistant chatbot, EDA script generation, and bug summarization and analysis. Our evaluations demonstrate that domain-adaptive pretraining of language models, can lead to superior performance in domain related downstream tasks compared to their base LLaMA2 counterparts, without degradations in generic capabilities. In particular, our largest model, ChipNeMo-70B, outperforms the highly capable GPT-4 on two of our use cases, namely engineering assistant chatbot and EDA scripts generation, while exhibiting competitive performance on bug summarization and analysis. These results underscore the potential of domain-specific customization for enhancing the effectiveness of large language models in specialized applications.