MHRC-Bench: A Multilingual Hardware Repository-Level Code Completion benchmark
Not on the map. Why?
Abstract (arXiv)
Large language models (LLMs) have achieved strong performance on code completion tasks in general-purpose programming languages. However, existing repository-level code completion benchmarks focus almost exclusively on software code and largely overlook hardware description languages. In this work, we present MHRC-Bench, consisting of MHRC-Bench-Train and MHRC-Bench-Eval, the first benchmark designed for multilingual hardware code completion at the repository level. Our benchmark targets completion tasks and covers three major hardware design coding styles. Each completion target is annotated with code-structure-level and hardware-oriented semantic labels derived from concrete syntax tree analysis. We conduct a comprehensive evaluation of models on MHRC-Bench-Eval. Comprehensive evaluation results and analysis demonstrate the effectiveness of MHRC-Bench.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 29 of its 39 references that are in the map. First posted on arXiv 7 Jan 2026.
Builds on
29 of 39 references.
References · 29
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
- 2024Qwen2.5-Coder Technical Reportcited by 1,151
- 2025OpenAI GPT-5 System Cardcited by 910
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
- 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 357
- 2023RepoBench: Benchmarking Repository-Level Code Auto-Completion Systemscited by 346
- 2023VerilogEval: Evaluating Large Language Models for Verilog Code Generationcited by 246
- 2023Using an LLM to Help With Code Understandingcited by 222
- 2023CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completioncited by 181
- 2023RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Modelcited by 143
17 more
- 2024Beyond Functional Correctness: Exploring Hallucinations in LLM-Generated Codecited by 106
- 2024CodexGraph: Bridging Large Language Models and Code Repositories via Code Graph Databasescited by 62
- 2024RLCoder: Reinforcement Learning for Repository-Level Code Completioncited by 44
- 2024GraphCoder: Enhancing Repository-Level Code Completion via Code Context Graph-based Retrieval and Language Modelcited by 40
- 2025ResearchCodeBench: Benchmarking LLMs on Implementing Novel Machine Learning Research Codecited by 34
- 2024REPOFUSE: Repository-Level Code Completion with Fused Dual Contextcited by 32
- 2024mHumanEval -- A Multilingual Benchmark to Evaluate Large Language Models for Code Generationcited by 32
- 2024RTL-Repo: A Benchmark for Evaluating LLMs on Large-Scale RTL Design Projectscited by 25
- 2024M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluationcited by 21
- 2024ExecRepoBench: Multi-level Executable Code Completion Evaluationcited by 19
- 2024AssertionBench: A Benchmark to Evaluate Large-Language Models for Assertion Generationcited by 16
- 2024Hierarchical Context Pruning: Optimizing Real-World Code Completion with Repository-Level Pretrained Code LLMscited by 13
- 2025Divergent Thoughts toward One Goal: LLM-based Multi-Agent Collaboration System for Electronic Design Automationcited by 12
- 2025HLS-Eval: A Benchmark and Framework for Evaluating LLMs on High-Level Synthesis Design Taskscited by 10
- 2025DeepRTL2: A Versatile Model for RTL-Related Taskscited by 10
- 2025RTLRepoCoder: Repository-Level RTL Code Completion through the Combination of Fine-Tuning and Retrieval Augmentationcited by 7
- 2025Enhancing Project-Specific Code Completion by Inferring Internal API Informationcited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.