# Structured Reinforcement Learning for Incentivized Stochastic Covert Optimization

arXiv: https://arxiv.org/abs/2405.07415 · HTML: https://mlsanity.com/paper/2405.07415/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 19 references that are in the map.
- First posted on arXiv 13 May 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 7 of 7)

- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Review](https://mlsanity.com/paper/2205.07855/) (2022)
- [Private Sequential Learning](https://mlsanity.com/paper/1805.02136/) (2018)
- [Finite-Time Convergent Algorithms for Time-Varying Distributed Optimization](https://mlsanity.com/paper/2210.03947/) (2022)
- [Learner-Private Convex Optimization](https://mlsanity.com/paper/2102.11976/) (2021)
- [Interval Dominance based Structural Results for Markov Decision Process](https://mlsanity.com/paper/2203.10618/) (2022)
- [Controlling Federated Learning for Covertness](https://mlsanity.com/paper/2308.08825/) (2023)

## Built on by (most cited first, 1 of 1)

- [Interacting Large Language Model Agents. Interpretable Models and Social Learning](https://mlsanity.com/paper/2411.01271/) (2024)

## Abstract (arXiv)

This paper studies how a stochastic gradient algorithm (SG) can be controlled to hide the estimate of the local stationary point from an eavesdropper. Such problems are of significant interest in distributed optimization settings like federated learning and inventory management. A learner queries a stochastic oracle and incentivizes the oracle to obtain noisy gradient measurements and perform SG. The oracle probabilistically returns either a noisy gradient of the function} or a non-informative measurement, depending on the oracle state and incentive. The learner's query and incentive are visible to an eavesdropper who wishes to estimate the stationary point. This paper formulates the problem of the learner performing covert optimization by dynamically incentivizing the stochastic oracle and obfuscating the eavesdropper as a finite-horizon Markov decision process (MDP). Using conditions for interval-dominance on the cost and transition probability structure, we show that the optimal policy for the MDP has a monotone threshold structure. We propose searching for the optimal stationary policy with the threshold structure using a stochastic approximation algorithm and a multi-armed bandit approach. The effectiveness of our methods is numerically demonstrated on a covert federated learning hate-speech classification task.