# Minimum information divergence of Q-functions for dynamic treatment resumes

arXiv: https://arxiv.org/abs/2211.08741 · HTML: https://mlsanity.com/paper/2211.08741/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 1 of its 29 references that are in the map.
- First posted on arXiv 16 Nov 2022.

## Builds on (most cited first, 1 of 1)

- [Estimating Time-Varying Causal Excursion Effect in Mobile Health with Binary Outcomes](https://mlsanity.com/paper/1906.00528/) (2019)

## Abstract (arXiv)

This paper aims at presenting a new application of information geometry to reinforcement learning focusing on dynamic treatment resumes. In a standard framework of reinforcement learning, a Q-function is defined as the conditional expectation of a reward given a state and an action for a single-stage situation. We introduce an equivalence relation, called the policy equivalence, in the space of all the Q-functions. A class of information divergence is defined in the Q-function space for every stage. The main objective is to propose an estimator of the optimal policy function by a method of minimum information divergence based on a dataset of trajectories. In particular, we discuss the γ-power divergence that is shown to have an advantageous property such that the γ-power divergence between policy-equivalent Q-functions vanishes. This property essentially works to seek the optimal policy, which is discussed in a framework of a semiparametric model for the Q-function. The specific choices of power index γ give interesting relationships of the value function, and the geometric and harmonic means of the Q-function. A numerical experiment demonstrates the performance of the minimum γ-power divergence method in the context of dynamic treatment regimes.