# A Reproducible Semantic Benchmark for Multivendor DSM-to-CLI Translation

arXiv: https://arxiv.org/abs/2606.20564 · HTML: https://mlsanity.com/paper/2606.20564/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 12 references that are in the map.
- First posted on arXiv 5 Apr 2026.

## Builds on (most cited first, 3 of 3)

- [A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions](https://mlsanity.com/paper/2412.19823/) (2024)
- [Large Language Models for Networking: Workflow, Advances and Challenges](https://mlsanity.com/paper/2404.12901/) (2024)
- [INTA: Intent-Based Translation for Network Configuration with LLM Agents](https://mlsanity.com/paper/2501.08760/) (2025)

## Built on by (most cited first, 1 of 1)

- [SoK: Semantic Decision Engines in Network Control Loops](https://mlsanity.com/paper/2610.06425/) (2026)

## Abstract (arXiv)

Translating high-level network intents into correct multivendor configurations remains a central challenge in network automation, as syntactically valid outputs may still violate the intended operational state. Despite recent advances in Large Language Models (LLMs), the field still lacks reproducible semantic benchmarks for rigorous cross-vendor evaluation. This paper presents a reproducible DSM-to-CLI semantic benchmark covering five cloud LLMs, three vendors, five representative use cases, and ten repeated runs per experimental cell under fixed judges and an explicit failure taxonomy. Our results show that semantic quality and operational reliability are orthogonal, vendor effects dominate use-case effects, and repeated-run dispersion strongly predicts vote instability, with Huawei VRP exposing failure modes hidden by aggregate metrics. These findings demonstrate that multivendor, repeated-execution semantic benchmarks are essential for scientifically rigorous comparison of LLM-based network configuration systems.