# SIMCODE: A Benchmark for Natural Language to ns-3 Network Simulation Code Generation

arXiv: https://arxiv.org/abs/2507.11014 · HTML: https://mlsanity.com/paper/2507.11014/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jul 2025.

## Built on by (most cited first, 2 of 2)

- [Can Large Language Models Implement Agent-Based Models? An ODD-based Replication Study](https://mlsanity.com/paper/2602.10140/) (2026)
- [PHITSBench: an execution-scored benchmark for AI-assisted PHITS radiation-transport input generation using natural language](https://mlsanity.com/paper/2607.09789/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in code generation across various domains. However, their effectiveness in generating simulation scripts for domain-specific environments like ns-3 remains underexplored. Despite the growing interest in automating network simulations, existing tools primarily focus on interactive automation over rigorous evaluation. To facilitate systematic evaluation, we introduce SIMCODE, the first benchmark to evaluate LLMs' ability to generate ns-3 simulation code from natural language. SIMCODE includes 400 tasks across introductory, intermediate, and advanced levels, with solutions and test cases. Using SIMCODE, we evaluate three prominent LLMs, Gemini-2.0, GPT-4.1, and Qwen-3, across six prompt techniques. Furthermore, investigating task-specific fine-tuning's impact reveals that while GPT-4.1 outperforms others, execution accuracy remains modest, with substantial room for improvement. Error analysis identifies missing headers and API mismatches as dominant failures. Nevertheless, SIMCODE provides a foundational step toward evaluating LLMs and research in domain-aware generative systems.