mlsanityData to 14 Oct 2026

Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QA

arXiv 2407.15353 ↗First posted 22 Jul 2024cs.CL cs.ARcited by 28

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval augmented generation (RAG) enhances the accuracy and reliability of generative AI models by sourcing factual information from external databases, which is extensively employed in document-grounded question-answering (QA) tasks. Off-the-shelf RAG flows are well pretrained on general-purpose documents, yet they encounter significant challenges when being applied to knowledge-intensive vertical domains, such as electronic design automation (EDA). This paper addresses such issue by proposing a customized RAG framework along with three domain-specific techniques for EDA tool documentation QA, including a contrastive learning scheme for text embedding model fine-tuning, a reranker distilled from proprietary LLM, and a generative LLM fine-tuned with high-quality domain corpus. Furthermore, we have developed and released a documentation QA evaluation benchmark, ORD-QA, for OpenROAD, an advanced RTL-to-GDSII design platform. Experimental results demonstrate that our proposed RAG flow and techniques have achieved superior performance on ORD-QA as well as on a commercial tool, compared with state-of-the-arts. The ORD-QA benchmark and the training dataset for our customized RAG flow are open-source at https://github.com/lesliepy99/RAG-EDA.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 30 of its 43 references that are in the map. First posted on arXiv 22 Jul 2024.

Builds on

30 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024The Power of Noise: Redefining Retrieval for RAG Systems26 Jan

Other references · 29

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  4. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  7. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  8. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  9. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  10. 2023LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compressioncited by 453
  11. 2024LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compressioncited by 348
  12. 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergycited by 346
17 more
  1. 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 252
  2. 2023VeriGen: A Large Language Model for Verilog Code Generationcited by 200
  3. 2023ChipNeMo: Domain-Adapted LLMs for Chip Designcited by 174
  4. 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 148
  5. 2023Chip-Chat: Challenges and Opportunities in Conversational Hardware Designcited by 140
  6. 2023ChatEDA: A Large Language Model Powered Autonomous Agent for EDAcited by 122
  7. 2022Benchmarking Large Language Models for Automated Verilog RTL Code Generationcited by 115
  8. 2023ChipGPT: How far are we from natural language hardware designcited by 105
  9. 2023RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Modelscited by 100
  10. 2024BetterV: Controlled Verilog Generation with Discriminative Guidancecited by 100
  11. 2023AutoChip: Automating HDL Generation Using LLM Feedbackcited by 83
  12. 2023GPT4AIGChip: Towards Next-Generation AI Accelerator Design Automation via Large Language Modelscited by 61
  13. 2022An Analysis of Fusion Functions for Hybrid Retrievalcited by 51
  14. 2024HDLdebugger: Streamlining HDL debugging with Large Language Modelscited by 26
  15. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  16. 2023Unlocking Hardware Security Assurance: The Potential of LLMscited by 9
  17. 2023iEDA: An Open-Source Intelligent Physical Implementation Toolkit and Librarycited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 12.

28 papers.

Sort:

By area:

  1. 2025A Survey of Research in Large Language Models for Electronic Design Automationcited by 41
  2. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  3. 2025AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMscited by 15
  4. 2025ORFS-agent: Tool-Using Agents for Chip Design Optimizationcited by 14
  5. 2025Revolution or Hype? Seeking the Limits of Large Models in Hardware Designcited by 9
  6. 2025Towards Global Retrieval Augmented Generation: A Benchmark for Corpus-Level Reasoningcited by 8
  7. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  8. 2024ChipAlign: Instruction Alignment in Large Language Models for Chip Design via Geodesic Interpolationcited by 6
  9. 2025MMCircuitEval: A Comprehensive Multimodal Circuit-Focused Benchmark for Evaluating LLMscited by 6
  10. 2024ORAssistant: A Custom RAG-based Conversational Assistant for OpenROADcited by 4
  11. 2025Large Language Models (LLMs) for Electronic Design Automation (EDA)cited by 3
  12. 2026MulTaBench: Benchmarking Multimodal Tabular Learning with Text and Imagecited by 2
16 more
  1. 2025BRIDGES: Bridging Graph Modality and Large Language Models within EDA Taskscited by 1
  2. 2025HDLxGraph: Bridging Large Language Models and HDL Repositories via HDL Graph Databasescited by 1
  3. 2026AnalogRetriever: Learning Cross-Modal Representations for Analog Circuit Retrievalcited by 1
  4. 2026H2MT: Semantic Hierarchy-Aware Hierarchical Memory Transformercited by 1
  5. 2026ZhuLong: Execution-Grounded LLM Agent for EDA Scripting with Offline API Self-Explorationcited by 1
  6. 2024A Retrieval-Augmented Generation Framework for Academic Literature Navigation in Data Sciencecited by 0
  7. 2025D2S-FLOW: Automated Parameter Extraction from Datasheets for SPICE Model Generation Using Large Language Modelscited by 0
  8. 2025Enhancing Technical Documents Retrieval for RAGcited by 0
  9. 2026Understand and Accelerate Memory Processing Pipeline for Large Language Model Inferencecited by 0
  10. 2026ChipLingo: A Systematic Training Framework for Large Language Models in EDAcited by 0
  11. 2026Verilog-Evolve: Feedback-Driven and Skill-Evolving Verilog Generationcited by 0
  12. 2026Agentic Electronic Design Automation: A Handoff Perspectivecited by 0
  13. 2026SCALE: Self-Supervised Constraint-Aware Layout GEneration for Local P&R DRV Fixing at Advanced Nodescited by 0
  14. 2026SynAct: A Reasoning-Acting Large Language Model Agent for Adaptive Synthesis Optimizationcited by 0
  15. 2026StateTune: Transforming LLM-Assisted EDA Flow Tuning into a Stateful, Closed-Loop Processcited by 0
  16. 2026SpecRead: A Benchmark for Measuring Whether Language Models Understand Hardware Specificationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026From Chunks to Functional Evidence: Function-Aware Retrieval for EDA Documentation QA
  2. 2024Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report
  3. 2025Evaluation of retrieval-based QA on QUEST-LOFT
  4. 2024Meta Knowledge for Retrieval Augmented Large Language Models
  5. 2025Retrieval-Augmented Generation in Industry: An Interview Study on Use Cases, Requirements, Challenges, and Evaluation
  6. 2024Retrieval Augmented Generation Systems: Automatic Dataset Creation, Evaluation and Boolean Agent Setup
  7. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation
  8. 2024Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective