# Multimodal Chip Physical Design Engineer Assistant

arXiv: https://arxiv.org/abs/2510.15872 · HTML: https://mlsanity.com/paper/2510.15872/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 2 Jul 2025.

## Builds on (most cited first, 29 of 29)

- [MiniCPM-V: A GPT-4V Level MLLM on Your Phone](https://mlsanity.com/paper/2408.01800/) (2024)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [MultiModal-GPT: A Vision and Language Model for Dialogue with Humans](https://mlsanity.com/paper/2305.04790/) (2023)
- [ChipNeMo: Domain-Adapted LLMs for Chip Design](https://mlsanity.com/paper/2311.00176/) (2023)
- [Chip-Chat: Challenges and Opportunities in Conversational Hardware Design](https://mlsanity.com/paper/2305.13243/) (2023)
- [RTLFixer: Automatically Fixing RTL Syntax Errors with Large Language Models](https://mlsanity.com/paper/2311.16543/) (2023)
- [Comprehensive Verilog Design Problems: A Next-Generation Benchmark Dataset for Evaluating Large Language Models and Agents on RTL Design and Verification](https://mlsanity.com/paper/2506.14074/) (2025)
- [CraftRTL: High-quality Synthetic Data Generation for Verilog Code Models with Correct-by-Construction Non-Textual Representations and Targeted Code Repair](https://mlsanity.com/paper/2409.12993/) (2024)
- [Towards Optimizing with Large Language Models](https://mlsanity.com/paper/2310.05204/) (2023)
- [Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models](https://mlsanity.com/paper/2403.06199/) (2024)
- [ScaleRTL: Scaling LLMs with Reasoning Data and Test-Time Compute for Accurate RTL Code Generation](https://mlsanity.com/paper/2506.05566/) (2025)
- [Code Less, Align More: Efficient LLM Fine-tuning for Code Generation with Data Pruning](https://mlsanity.com/paper/2407.05040/) (2024)
- [Genetic Instruct: Scaling up Synthetic Generation of Coding Instructions for Large Language Models](https://mlsanity.com/paper/2407.21077/) (2024)
- [Text-centric Alignment for Multi-Modality Learning](https://mlsanity.com/paper/2402.08086/) (2024)
- [Toward More Generalized Malicious URL Detection Models](https://mlsanity.com/paper/2202.10027/) (2022)
- [Differential Good Arm Identification](https://mlsanity.com/paper/2303.07154/) (2023)
- [Fast online inference for nonlinear contextual bandit based on Generative Adversarial Network](https://mlsanity.com/paper/2202.08867/) (2022)
- [AutoML-GPT: Large Language Model for AutoML](https://mlsanity.com/paper/2309.01125/) (2023)
- [lil'HDoC: An Algorithm for Good Arm Identification under Small Threshold Gap](https://mlsanity.com/paper/2401.15879/) (2024)
- [Enhance Modality Robustness in Text-Centric Multimodal Alignment with Adversarial Prompting](https://mlsanity.com/paper/2408.09798/) (2024)
- [Enhance the Robustness of Text-Centric Multimodal Alignments](https://mlsanity.com/paper/2407.05036/) (2024)
- [Uncertainty Profiles for LLMs: Uncertainty Source Decomposition and Adaptive Model-Metric Selection](https://mlsanity.com/paper/2505.07309/) (2025)
- [Circuit as Set of Points](https://mlsanity.com/paper/2310.17418/) (2023)
- [PSGText: Stroke-Guided Scene Text Editing with PSP Module](https://mlsanity.com/paper/2310.13366/) (2023)
- [LinearAPT: An Adaptive Algorithm for the Fixed-Budget Thresholding Linear Bandit Problem](https://mlsanity.com/paper/2403.06230/) (2024)
- [Benchmarking Large Language Model Uncertainty for Prompt Optimization](https://mlsanity.com/paper/2409.10044/) (2024)
- [Optimizing Predictive AI in Physical Design Flows with Mini Pixel Batch Gradient Descent](https://mlsanity.com/paper/2402.06034/) (2024)
- [Generalizing Large Language Model Usability Across Resource-Constrained](https://mlsanity.com/paper/2505.17040/) (2025)

## Built on by (most cited first, 1 of 1)

- [Exploring the Agentic Frontier of Verilog Code Generation](https://mlsanity.com/paper/2603.19347/) (2026)

## Abstract (arXiv)

Modern chip physical design relies heavily on Electronic Design Automation (EDA) tools, which often struggle to provide interpretable feedback or actionable guidance for improving routing congestion. In this work, we introduce a Multimodal Large Language Model Assistant (MLLMA) that bridges this gap by not only predicting congestion but also delivering human-interpretable design suggestions. Our method combines automated feature generation through MLLM-guided genetic prompting with an interpretable preference learning framework that models congestion-relevant tradeoffs across visual, tabular, and textual inputs. We compile these insights into a "Design Suggestion Deck" that surfaces the most influential layout features and proposes targeted optimizations. Experiments on the CircuitNet benchmark demonstrate that our approach outperforms existing models on both accuracy and explainability. Additionally, our design suggestion guidance case study and qualitative analyses confirm that the learned preferences align with real-world design principles and are actionable for engineers. This work highlights the potential of MLLMs as interactive assistants for interpretable and context-aware physical design optimization.