α-GAN by R'enyi Cross Entropy

This paper proposes α-GAN, a generative adversarial network using R'enyi measures. The value function is formulated, by R'enyi cross entropy, as an expected certainty measure incurred by the discriminator's soft decision as to where the sample is from, true population or the generator. The discriminator tries to maximize the R'enyi certainty about sample source, while the generator wants to reduce it by injecting fake samples. This forms a min-max problem with the solution parameterized by the R'enyi order α. This α-GAN reduces to vanilla GAN at α = 1, where the value function is exactly the binary cross entropy. The optimization of α-GAN is over probability (vector) space. It is shown that the gradient is exponentially enlarged when R'enyi order is in the range α ∈ (0,1). This makes convergence faster, which is verified by experimental results. A discussion shows that choosing α ∈ (0,1) may be able to solve some common problems, e.g., vanishing gradient. A following observation reveals that this range has not been fully explored in the existing R'enyi version GANs.