Acceleration for Compressed Gradient Descent in Distributed and Federated Optimization

Due to the high communication cost in distributed and federated learning problems, methods relying on compression of communicated messages are becoming increasingly popular. While in other contexts the best performing gradient-type methods invariably rely on some form of acceleration/momentum to reduce the number of iterations, there are no methods which combine the benefits of both gradient compression and acceleration. In this paper, we remedy this situation and propose the first accelerated compressed gradient descent (ACGD) methods. In the single machine regime, we prove that ACGD enjoys the rate OBig((1+ω)√fracLμlog frac1εBig) for μ-strongly convex problems and OBig((1+ω)√fracLεBig) for convex problems, respectively, where ω is the compression parameter. Our results improve upon the existing non-accelerated rates OBig((1+ω)fracLμlog frac1εBig) and OBig((1+ω)fracLεBig), respectively, and recover the optimal rates of accelerated gradient descent as a special case when no compression (ω=0) is applied. We further propose a distributed variant of ACGD (called ADIANA) and prove the convergence rate widetildeOBig(ω+√fracLμ+√big(fracωn+√fracωnbig)fracω LμBig), where n is the number of devices/workers and widetildeO hides the logarithmic factor log frac1ε. This improves upon the previous best result widetildeOBig(ω + fracLμ+fracω Lnμ Big) achieved by the DIANA method of Mishchenko et al. (2019). Finally, we conduct several experiments on real-world datasets which corroborate our theoretical results and confirm the practical superiority of our accelerated methods.