mlsanityData to 14 Oct 2026

CreativEval: Evaluating Creativity of LLM-Based Hardware Code Generation

arXiv 2404.08806 ↗First posted 12 Apr 2024cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have proved effective and efficient in generating code, leading to their utilization within the hardware design process. Prior works evaluating LLMs' abilities for register transfer level code generation solely focus on functional correctness. However, the creativity associated with these LLMs, or the ability to generate novel and unique solutions, is a metric not as well understood, in part due to the challenge of quantifying this quality. To address this research gap, we present CreativeEval, a framework for evaluating the creativity of LLMs within the context of generating hardware designs. We quantify four creative sub-components, fluency, flexibility, originality, and elaboration, through various prompting and post-processing techniques. We then evaluate multiple popular LLMs (including GPT models, CodeLlama, and VeriGen) upon this creativity metric, with results indicating GPT-3.5 as the most creative model in generating hardware designs.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 23 of its 36 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

23 of 36 references.

References · 23

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023WizardCoder: Empowering Code Large Language Models with Evol-Instructcited by 733
  6. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 420
  7. 2022Using cognitive psychology to understand GPT-3cited by 398
  8. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  9. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  10. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  11. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  12. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
11 more
  1. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  2. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  3. 2023AutoChip: Automating HDL Generation Using LLM Feedbackcited by 83
  4. 2022Putting GPT-3's Creativity to the (Alternative Uses) Testcited by 62
  5. 2023Automating Customer Service using LangChain: Building custom open-source GPT Chatbot for organizationscited by 61
  6. 2020DAVE: Deriving Automatically Verilog from Englishcited by 48
  7. 2024Make Every Move Count: LLM-based High-Quality RTL Code Generation Using MCTScited by 43
  8. 2023LLM4EDA: Emerging Progress in Large Language Models for Electronic Design Automationcited by 40
  9. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  10. 2023Do large language models resemble humans in language use?cited by 34
  11. 2021GNN4IP: Graph Neural Network for Hardware Intellectual Property Piracy Detectioncited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 8; 2026: 6.

19 papers.

Sort:

By area:

  1. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  2. 2024Benchmarking Language Model Creativity: A Case Study on Code Generationcited by 40
  3. 2026CreativeBench: Benchmarking and Enhancing Machine Creativity via Self-Evolving Challengescited by 9
  4. 2025Large Language Model for Verilog Code Generation: Literature Review and the Road Aheadcited by 8
  5. 2024LLMPirate: LLMs for Black-box Hardware IP Piracycited by 7
  6. 2025HypoSpace: A Diagnostic Benchmark for Set-Valued Hypothesis Generation under Underdetermination and Sublinear Coverage Boundscited by 6
  7. 2025Large Language Model Sourcing: A Surveycited by 5
  8. 2024Artificial Intelligence as the New Hacker: Developing Agents for Offensive Securitycited by 4
  9. 2025What Shapes a Creative Machine Mind? Comprehensively Benchmarking Creativity in Foundation Modelscited by 2
  10. 2026Automated Creativity Evaluation of Language Models Across Open-Ended Taskscited by 2
  11. 2025Proof2Silicon: Prompt Repair for Verified Code and Hardware Generation via Reinforcement Learningcited by 1
  12. 2026RTL-BenchLS: A Large-Scale Benchmark for RTL Reasoning and Generation with Large Language Modelscited by 1
7 more
  1. 2026Can LLM Agents Discover? Evaluating Creativity on ML Engineering Taskscited by 1
  2. 2024Initial Development and Evaluation of the Creative Artificial Intelligence through Recurring Developments and Determinations (CAIRDD) Systemcited by 0
  3. 2025Lorecast: Layout-Aware Performance and Power Forecasting from Natural Languagecited by 0
  4. 2025ML For Hardware Design Interpretability: Challenges and Opportunitiescited by 0
  5. 2025PEFA-AI: Advancing Open-source LLMs for RTL generation using Progressive Error Feedback Agentic-AIcited by 0
  6. 2026How Creative Are Large Language Models in Generating Molecules?cited by 0
  7. 2026Agentic Electronic Design Automation: A Handoff Perspectivecited by 0