Adam Converges Without Any Modification On Update Rules

Adam is the default algorithm for training neural networks, including large language models (LLMs). However, citetreddi2019convergence provided an example that Adam diverges, raising concerns for its deployment in AI model training. We identify a key mismatch between the divergence example and practice: citetreddi2019convergence pick the problem after picking the hyperparameters of Adam, i.e., (β1,β2); while practical applications often fix the problem first and then tune (β1,β2). In this work, we prove that Adam converges with proper problem-dependent hyperparameters. First, we prove that Adam converges when β2 is large and β1 < √β2. Second, when β2 is small, we point out a region of (β1,β2) combinations where Adam can diverge to infinity. Our results indicate a phase transition for Adam from divergence to convergence when changing the (β1, β2) combination. To our knowledge, this is the first phase transition in (β1,β2) 2D-plane reported in the literature, providing rigorous theoretical guarantees for Adam optimizer. We further point out that the critical boundary (β1^*, β2^*) is problem-dependent, and particularly, dependent on batch size. This provides suggestions on how to tune β1 and β2: when Adam does not work well, we suggest tuning up β2 inversely with batch size to surpass the threshold β2^*, and then trying β1< √β2. Our suggestions are supported by reports from several empirical studies, which observe improved LLM training performance when applying them.