# HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections

arXiv: https://arxiv.org/abs/2007.05891 · HTML: https://mlsanity.com/paper/2007.05891/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jul 2020.

## Built on by (most cited first, 3 of 3)

- [UL2: Unifying Language Learning Paradigms](https://mlsanity.com/paper/2205.05131/) (2022)
- [HyperPrompt: Prompt-based Task-Conditioning of Transformers](https://mlsanity.com/paper/2203.00759/) (2022)
- [Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey](https://mlsanity.com/paper/2308.08234/) (2023)

## Abstract (arXiv)

Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning a single multi-task model that is able to do well for all the tasks has been a challenging and yet attractive proposition. In this paper, we propose textscHyperGrid, a new approach for highly effective multi-task learning. The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks. In order to construct the proposed hypernetwork, our method learns the interactions and composition between a global (task-agnostic) state and a local task-specific state. We apply our proposed textscHyperGrid on the current state-of-the-art T5 model, demonstrating strong performance across the GLUE and SuperGLUE benchmarks when using only a single multi-task model. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.