# PICBench: Benchmarking LLMs for Photonic Integrated Circuits Design

arXiv: https://arxiv.org/abs/2502.03159 · HTML: https://mlsanity.com/paper/2502.03159/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 5 Feb 2025.

## Builds on (most cited first, 7 of 7)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [VerilogEval: Evaluating Large Language Models for Verilog Code Generation](https://mlsanity.com/paper/2309.07544/) (2023)
- [RTLLM: An Open-Source Benchmark for Design RTL Generation with Large Language Model](https://mlsanity.com/paper/2308.05345/) (2023)
- [ChatEDA: A Large Language Model Powered Autonomous Agent for EDA](https://mlsanity.com/paper/2308.10204/) (2023)
- [SPICEPilot: Navigating SPICE Code Generation and Simulation with AI Guidance](https://mlsanity.com/paper/2410.20553/) (2024)
- [Empowering high-dimensional optical fiber communications with integrated photonic processors](https://mlsanity.com/paper/2311.05282/) (2023)

## Built on by (most cited first, 5 of 5)

- [AI Agents for Photonic Integrated Circuit Design Automation](https://mlsanity.com/paper/2508.14123/) (2025)
- [Agentic Electronic Design Automation: A Handoff Perspective](https://mlsanity.com/paper/2606.19795/) (2026)
- [From Pixels to PCells: A Neurosymbolic Approach to Photonic Component Creation](https://mlsanity.com/paper/2608.00084/) (2026)
- [PICopilot: An LLM-based Agentic Framework for Assisting Photonic Integrated Circuit Design via Script Generation](https://mlsanity.com/paper/2608.01791/) (2026)
- [PICasso: An AI-Enabled Design Framework for Autonomous Optimization of Silicon Photonic Devices](https://mlsanity.com/paper/2608.26113/) (2026)

## Abstract (arXiv)

While large language models (LLMs) have shown remarkable potential in automating various tasks in digital chip design, the field of Photonic Integrated Circuits (PICs)-a promising solution to advanced chip designs-remains relatively unexplored in this context. The design of PICs is time-consuming and prone to errors due to the extensive and repetitive nature of code involved in photonic chip design. In this paper, we introduce PICBench, the first benchmarking and evaluation framework specifically designed to automate PIC design generation using LLMs, where the generated output takes the form of a netlist. Our benchmark consists of dozens of meticulously crafted PIC design problems, spanning from fundamental device designs to more complex circuit-level designs. It automatically evaluates both the syntax and functionality of generated PIC designs by comparing simulation outputs with expert-written solutions, leveraging an open-source simulator. We evaluate a range of existing LLMs, while also conducting comparative tests on various prompt engineering techniques to enhance LLM performance in automated PIC design. The results reveal the challenges and potential of LLMs in the PIC design domain, offering insights into the key areas that require further research and development to optimize automation in this field. Our benchmark and evaluation code is available at https://github.com/PICDA/PICBench.