# DynaMark: A Reinforcement Learning Framework for Dynamic Watermarking in Industrial Machine Tool Controllers

arXiv: https://arxiv.org/abs/2508.21797 · HTML: https://mlsanity.com/paper/2508.21797/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 10 of its 59 references that are in the map.
- First posted on arXiv 29 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 10 of 10)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [The problem with DDPG: understanding failures in deterministic environments with sparse rewards](https://mlsanity.com/paper/1911.11679/) (2019)
- [Dynamic Watermarking: Active Defense of Networked Cyber-Physical Systems](https://mlsanity.com/paper/1606.08741/) (2016)
- [An Online Approach to Physical Watermark Design](https://mlsanity.com/paper/1911.01868/) (2019)
- [A Survey of Cybersecurity of Digital Manufacturing](https://mlsanity.com/paper/2006.05042/) (2020)
- [Detecting Generalized Replay Attacks via Time-Varying Dynamic Watermarking](https://mlsanity.com/paper/1909.08111/) (2019)
- [Dynamic Watermarking for General LTI Systems](https://mlsanity.com/paper/1703.07760/) (2017)
- [Sequential detection of Replay attacks](https://mlsanity.com/paper/2012.10748/) (2020)
- [Secure Control of Networked Control Systems Using Dynamic Watermarking](https://mlsanity.com/paper/2108.11572/) (2021)

## Abstract (arXiv)

Industry 4.0's highly networked Machine Tool Controllers (MTCs) are prime targets for replay attacks that use outdated sensor data to manipulate actuators. Dynamic watermarking can reveal such tampering, but current schemes assume linear-Gaussian dynamics and use constant watermark statistics, making them vulnerable to the time-varying, partly proprietary behavior of MTCs. We close this gap with DynaMark, a reinforcement learning framework that models dynamic watermarking as a Markov decision process (MDP). It learns an adaptive policy online that dynamically adapts the covariance of a zero-mean Gaussian watermark using available measurements and detector feedback, without needing system knowledge. DynaMark maximizes a unique reward function balancing control performance, energy consumption, and detection confidence dynamically. We develop a Bayesian belief updating mechanism for real-time detection confidence in linear systems. This approach, independent of specific system assumptions, underpins the MDP for systems with linear dynamics. On a Siemens Sinumerik 828D controller digital twin, DynaMark achieves a reduction in watermark energy by 70% while preserving the nominal trajectory, compared to constant variance baselines. It also maintains an average detection delay equivalent to one sampling interval. A physical stepper-motor testbed validates these findings, rapidly triggering alarms with less control performance decline and exceeding existing benchmarks.