# Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models

arXiv: https://arxiv.org/abs/2405.16560 · HTML: https://mlsanity.com/paper/2405.16560/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2024.

## Built on by (most cited first, 12 of 12)

- [Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent](https://mlsanity.com/paper/2501.01230/) (2025)
- [Selective Task Group Updates for Multi-Task Optimization](https://mlsanity.com/paper/2502.11986/) (2025)
- [Adaptive Defense against Harmful Fine-Tuning for Large Language Models via Bayesian Data Scheduler](https://mlsanity.com/paper/2510.27172/) (2025)
- [Weakly Supervised Temporal Action Localization via Dual-Prior Collaborative Learning Guided by Multimodal Large Language Models](https://mlsanity.com/paper/2411.08466/) (2024)
- [When Data-Free Knowledge Distillation Meets Non-Transferable Teacher: Escaping Out-of-Distribution Trap is All You Need](https://mlsanity.com/paper/2507.04119/) (2025)
- [Sparse Task Vector Mixup with Hypernetworks for Efficient Knowledge Transfer in Whole-Slide Image Prognosis](https://mlsanity.com/paper/2603.10526/) (2026)
- [PACT: Preserving Anchored Cores in Task-vectors for Model Merging](https://mlsanity.com/paper/2606.18627/) (2026)
- [Weight Weaving: Parameter Pooling for Data-Free Model Merging](https://mlsanity.com/paper/2510.13921/) (2025)
- [AI Security in the Foundation Model Era: A Comprehensive Survey from a Unified Perspective](https://mlsanity.com/paper/2603.24857/) (2026)
- [Dynamic Model Merging Made Slim](https://mlsanity.com/paper/2605.18904/) (2026)
- [STARS: Spike Tail-Aware Relational Synthesis for ANN-to-SNN Data-Free Knowledge Distillation](https://mlsanity.com/paper/2605.27409/) (2026)
- [Semantic-Aware Task Clustering for Constructive and Cooperative Multi-Tasking](https://mlsanity.com/paper/2607.21426/) (2026)

## Abstract (arXiv)

Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data, enabling the rapid adaptation to new unseen tasks. Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts. In this paper, we empirically and theoretically identify and analyze the model heterogeneity in DFML. We find that model heterogeneity introduces a heterogeneity-homogeneity trade-off, where homogeneous models reduce task conflicts but also increase the overfitting risk. Balancing this trade-off is crucial for learning shared representations across tasks. Based on our findings, we propose Task Groupings Regularization that benefits from model heterogeneity by grouping and aligning conflicting tasks. Specifically, we embed pre-trained models into a task space to compute dissimilarity, and group heterogeneous models together based on this measure. Then, we introduce implicit gradient regularization within each group to mitigate potential conflicts. By encouraging a gradient direction suitable for all tasks, the meta-model captures shared representations that generalize across tasks. Comprehensive experiments showcase the superiority of our approach in multiple benchmarks, effectively tackling the model heterogeneity in challenging multi-domain and multi-architecture scenarios.