Rethinking Gradient Modulation in Multimodal Regression

Even balanced multimodal learning methods do not consistently translate additional modalities into better regression performance. To understand this limitation, we revisit the optimization mechanism of balanced multimodal learning, using MMPareto as a representative case. We reveal a previously overlooked issue: MMPareto uses a fixed gradient modulation strength throughout training, while different training stages favor different strengths; an inappropriate modulation strength can instead hinder subsequent optimization. To address this issue, we propose SGM, which adapts the modulation strength according to the current training behavior. We provide theoretical and empirical analyses to characterize and validate the modulation decisions made by SGM. We further build a two-stage multimodal regression framework that integrates AM for unimodal regression representation learning and SGM for adaptive multimodal optimization. Extensive experiments under the same total training budget demonstrate consistent improvements over strong unimodal and multimodal baselines. Our code is provided in the supplementary material.