# An Index Policy Based on Sarsa and Q-learning for Heterogeneous Smart Target Tracking

arXiv: https://arxiv.org/abs/2402.12015 · HTML: https://mlsanity.com/paper/2402.12015/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 67 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 14 of 14)

- [Whittle index based Q-learning for restless bandits with average reward](https://mlsanity.com/paper/2004.14427/) (2020)
- [Learn to Intervene: An Adaptive Learning Policy for Restless Bandits in Application to Preventive Healthcare](https://mlsanity.com/paper/2105.07965/) (2021)
- [NeurWIN: Neural Whittle Index Network For Restless Bandits Via Deep RL](https://mlsanity.com/paper/2110.02128/) (2021)
- [Optimistic Whittle Index Policy: Online Learning for Restless Bandits](https://mlsanity.com/paper/2205.15372/) (2022)
- [Deep Reinforcement Learning Control for Radar Detection and Tracking in Congested Spectral Environments](https://mlsanity.com/paper/2006.13173/) (2020)
- [A verification theorem for threshold-indexability of real-state discounted restless bandits](https://mlsanity.com/paper/1902.09318/) (2019)
- [When are Kalman-filter restless bandits indexable?](https://mlsanity.com/paper/1509.04541/) (2015)
- [Learning Augmented Index Policy for Optimal Service Placement at the Network Edge](https://mlsanity.com/paper/2101.03641/) (2021)
- [Variational Nonlinear Kalman Filtering with Unknown Process Noise Covariance](https://mlsanity.com/paper/2305.03914/) (2023)
- [A Restless Bandit Model for Resource Allocation, Competition and Reservation](https://mlsanity.com/paper/1804.02100/) (2018)
- [Joint Waveform and Receiver Design for Co-Channel Hybrid Active-Passive Sensing with Timing Uncertainty](https://mlsanity.com/paper/1908.11016/) (2019)
- [Energy-Efficient Job-Assignment Policy with Asymptotically Guaranteed Performance Deviation](https://mlsanity.com/paper/1910.06567/) (2019)
- [A Restless Bandit Model for Energy-Efficient Job Assignments in Server Farms](https://mlsanity.com/paper/2112.06275/) (2021)
- [Timely Target Tracking: Distributed Updating in Cognitive Radar Networks](https://mlsanity.com/paper/2307.12936/) (2023)

## Abstract (arXiv)

In solving the non-myopic radar scheduling for multiple smart target tracking within an active and passive radar network, we need to consider both short-term enhanced tracking performance and a higher probability of target maneuvering in the future with active tracking. Acquiring the long-term tracking performance while scheduling the beam resources of active and passive radars poses a challenge. To address this challenge, we model this problem as a Markov decision process consisting of parallel restless bandit processes. Each bandit process is associated with a smart target, of which the estimation state evolves according to different discrete dynamic models for different actions - whether or not the target is being tracked. The discrete state is defined by the dynamic mode. The problem exhibits the curse of dimensionality, where optimal solutions are in general intractable. We resort to heuristics through the famous restless multi-armed bandit techniques. It follows with efficient scheduling policies based on the indices that are real numbers representing the marginal rewards of taking different actions. For the inevitable practical case with unknown transition matrices, we propose a new method that utilizes the forward Sarsa and backward Q-learning to approximate the indices through adapting the state-action value functions, or equivalently the Q-functions, and propose a new policy, namely ISQ, aiming to maximize the long-term tracking rewards. Numerical results demonstrate that the proposed ISQ policy outperforms conventional Q-learning-based methods and rapidly converges to the well-known Whittle index policy with revealed state transition models, which is considered the benchmark.