# Beyond touch-based human-machine interface: Control your machines in natural language by utilizing large language models and OPC UA

arXiv: https://arxiv.org/abs/2510.11300 · HTML: https://mlsanity.com/paper/2510.11300/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Builds on (most cited first, 10 of 10)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023)
- [gpt-oss-120b & gpt-oss-20b Model Card](https://mlsanity.com/paper/2508.10925/) (2025)
- [A Survey on Neural Speech Synthesis](https://mlsanity.com/paper/2106.15561/) (2021)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Agents4PLC: Automating Closed-loop PLC Code Generation and Verification in Industrial Control Systems using LLM-based Agents](https://mlsanity.com/paper/2410.14209/) (2024)
- [ChatGPT for PLC/DCS Control Logic Generation](https://mlsanity.com/paper/2305.15809/) (2023)
- [Natural User Interfaces: Trend in Virtual Interaction](https://mlsanity.com/paper/1405.0101/) (2014)
- [From Clicks to Conversations: Evaluating the Effectiveness of Conversational Agents in Statistical Analysis](https://mlsanity.com/paper/2502.08114/) (2025)

## Built on by (most cited first, 1 of 1)

- [LLM-Driven Multi-Agent Control for Skill-Based Smart Manufacturing](https://mlsanity.com/paper/2610.01364/) (2026)

## Abstract (arXiv)

This paper proposes an agent-based approach toward a more natural interface between humans and machines. Large language models equipped with tools and the communication standard OPC UA are utilized to control machines in natural language. Instead of touch interaction, which is currently the state-of-the-art medium for interaction in operations, the proposed approach enables operators to talk or text with machines. This allows commands such as 'Please decrease the temperature by 20 % in machine 1 and start the cleaning operation in machine 2.' The large language model receives the user input and selects one of three predefined tools that connect to an OPC UA server and either change or read the value of a node. Afterwards, the result of the tool execution is passed back to the language model, which then provides a final response to the user. The approach is universally designed and can therefore be applied to any machine that supports the OPC UA standard. The large language model is neither fine-tuned nor requires training data, only the relevant machine credentials and a parameter dictionary are included within the system prompt. The tool-calling ability and their design is evaluated on a demonstrator setup with a Siemens S7-1500 programmable logic controller with four machine parameters. Fifty synthetically generated commands on five different models were tested and the results demonstrate high success rate, with proprietary GPT-5 models achieving accuracies between 96.0 % and 98.0 %, and open-weight models reaching up to 90.0 %. Afterwards the approach was transferred to a deployed spay-coating machine. The proposed concept is supposed to contribute in advancing natural interaction in industrial human-machine interfaces.