Scalable Distributed Stochastic Optimization via Bidirectional Compression: Beyond Pessimistic Limits

In centralized, distributed, and federated learning with stochastic gradients and n workers, it was recently shown that it is infeasible to find an varepsilon-stationary point faster than tildeOmegaleft(minleftfracd kappa L Deltavarepsilon + frach L Deltavarepsilon + frach σ2 L Deltan varepsilon2,; frach σ2 L Deltavarepsilon2 + frach L Deltavarepsilonrightright) seconds in both homogeneous and heterogeneous settings under standard assumptions: L-smoothness, σ2-bounded unbiased stochastic gradients, and lower boundedness of the function, i.e., f(x) ≥ f^* for all x ∈ ℝd, where Delta = f(x0) - f^*, h is the computation time, kappa is the communication speed between the workers and the server, and d is the dimension of the iterates and gradients. This result is pessimistic since it does not allow a complexity in which both fracd kappa L Deltavarepsilon and frach σ2 L Deltavarepsilon2 improve with n, even when using random sparsification techniques; moreover, this lower bound can be matched by either non-distributed SGD or vanilla Synchronous SGD, which reduces the impact of recent progress in the design of compression-based methods. In this work, we challenge this limitation and propose new compressed methods, Inkheart SGD and M4, and show that under an additional structural assumption, which is necessary due to the lower bound and which does not restrict the class of considered problems, we achieve new state-of-the-art time complexities that break this pessimistic barrier and allow scaling with the number of workers n.