Learning to Orchestrate Vision Foundation Models for Multi-Task Dense Prediction

Vision foundation models (VFMs) exhibit complementary strengths shaped by their pretraining objectives. Yet prevailing methods for multi-task dense prediction still train an entire backbone, either by fine-tuning it under multi-task supervision or by distilling multiple VFMs in an additional stage. We ask whether downstream learning can instead compose the frozen representations already available in foundation models. Dense tasks require composite representations that no individual expert provides alone. Realizing them is difficult: simple fusion yields only marginal gains over the best single expert, while learned routing tends to collapse toward candidates that are strong at initialization, starving newly initialized composers of training signal. We present COVE, which constructs pairwise composite candidates through Synergy Composers and routes among raw and composite candidates with a Task-Conditioned Router. To prevent this collapse, COVE combines Gaussian logit perturbation for exploration with counterfactual supervision that selectively increases under-credited routing allocations. On NYUD-v2 and PASCAL-Context, COVE matches or surpasses ViT-L-based methods on most tasks using a smaller frozen encoder pool and roughly half the computation of recent VFM-based competitors, while exceeding the best single frozen expert on every task.