# Evaluating the effectiveness of LLM-based interoperability

arXiv: https://arxiv.org/abs/2510.23893 · HTML: https://mlsanity.com/paper/2510.23893/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 36 references that are in the map.
- First posted on arXiv 27 Oct 2025.

## Builds on (most cited first, 8 of 8)

- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching](https://mlsanity.com/paper/2303.16756/) (2023)
- [Exploring Large Language Models for Ontology Alignment](https://mlsanity.com/paper/2309.07172/) (2023)
- [Can LLMs Generate Architectural Design Decisions? -An Exploratory Empirical study](https://mlsanity.com/paper/2403.01709/) (2024)
- [Towards Evaluation Guidelines for Empirical Studies involving LLMs](https://mlsanity.com/paper/2411.07668/) (2024)
- [FhGenie: A Custom, Confidentiality-preserving Chat AI for Corporate and Scientific Use](https://mlsanity.com/paper/2403.00039/) (2024)
- [Systems Interoperability Types: A Tertiary Study](https://mlsanity.com/paper/2310.19999/) (2023)

## Built on by (most cited first, 1 of 1)

- [SAGAI-MID: A Generative AI-Driven Middleware for Dynamic Runtime Interoperability](https://mlsanity.com/paper/2603.28731/) (2026)

## Abstract (arXiv)

Background: Systems of systems are becoming increasingly dynamic and heterogeneous, and this adds pressure on the long-standing challenge of interoperability. Besides its technical aspect, interoperability has also an economic side, as development time efforts are required to build the interoperability artifacts. Objectives: With the recent advances in the field of large language models (LLMs), we aim at analyzing the effectiveness of LLM-based strategies to make systems interoperate autonomously, at runtime, without human intervention. Method: We selected 13 open source LLMs and curated four versions of a dataset in the agricultural interoperability use case. We performed three runs of each model with each version of the dataset, using two different strategies. Then we compared the effectiveness of the models and the consistency of their results across multiple runs. Results: qwen2.5-coder:32b was the most effective model using both strategies DIRECT (average pass@1 >= 0.99) and CODEGEN (average pass@1 >= 0.89) in three out of four dataset versions. In the fourth dataset version, which included an unit conversion, all models using the strategy DIRECT failed, whereas using CODEGEN qwen2.5-coder:32b succeeded with an average pass@1 = 0.75. Conclusion: Some LLMs can make systems interoperate autonomously. Further evaluation in different domains is recommended, and further research on reliability strategies should be conducted.