# OmniNet: A unified architecture for multi-modal multi-task learning

arXiv: https://arxiv.org/abs/1907.07804 · HTML: https://mlsanity.com/paper/1907.07804/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jul 2019.

## Built on by (most cited first, 25 of 25)

- [Multimodal Learning with Transformers: A Survey](https://mlsanity.com/paper/2206.06488/) (2022)
- [UniT: Multimodal Multitask Learning with a Unified Transformer](https://mlsanity.com/paper/2102.10772/) (2021)
- [OmniVec: Learning robust representations with cross modal sharing](https://mlsanity.com/paper/2311.05709/) (2023)
- [Multi-modal Machine Learning in Engineering Design: A Review and Future Directions](https://mlsanity.com/paper/2302.10909/) (2023)
- [Towards Versatile Embodied Navigation](https://mlsanity.com/paper/2210.16822/) (2022)
- [Integrating Language Guidance into Vision-based Deep Metric Learning](https://mlsanity.com/paper/2203.08543/) (2022)
- [Towards a Unified Foundation Model: Jointly Pre-Training Transformers on Unpaired Images and Text](https://mlsanity.com/paper/2112.07074/) (2021)
- [OmniVec2 -- A Novel Transformer based Network for Large Scale Multimodal and Multitask Learning](https://mlsanity.com/paper/2507.13364/) (2025)
- [Supervised Fine-tuning in turn Improves Visual Foundation Models](https://mlsanity.com/paper/2401.10222/) (2024)
- [Answer-Me: Multi-Task Open-Vocabulary Visual Question Answering](https://mlsanity.com/paper/2205.00949/) (2022)
- [Multi-Task Meta Learning: learn how to adapt to unseen tasks](https://mlsanity.com/paper/2210.06989/) (2022)
- [Harnessing the Power of Multi-Task Pretraining for Ground-Truth Level Natural Language Explanations](https://mlsanity.com/paper/2212.04231/) (2022)
- [MINOTAUR: Multi-task Video Grounding From Multimodal Queries](https://mlsanity.com/paper/2302.08063/) (2023)
- [GRID: A Platform for General Robot Intelligence Development](https://mlsanity.com/paper/2310.00887/) (2023)
- [Software/Hardware Co-design for Multi-modal Multi-task Learning in Autonomous Systems](https://mlsanity.com/paper/2104.04000/) (2021)
- [Exceeding the Limits of Visual-Linguistic Multi-Task Learning](https://mlsanity.com/paper/2107.13054/) (2021)
- [Learning More May Not Be Better: Knowledge Transferability in Vision and Language Tasks](https://mlsanity.com/paper/2208.10758/) (2022)
- [DynaShare: Task and Instance Conditioned Parameter Sharing for Multi-Task Learning](https://mlsanity.com/paper/2305.17305/) (2023)
- [ALFEE: Adaptive Large Foundation Model for EEG Representation](https://mlsanity.com/paper/2505.06291/) (2025)
- [ASC me to Do Anything: Multi-task Training for Embodied AI](https://mlsanity.com/paper/2202.06987/) (2022)
- [Challenges and Opportunities of Using Transformer-Based Multi-Task Learning in NLP Through ML Lifecycle: A Survey](https://mlsanity.com/paper/2308.08234/) (2023)
- [Few-shot Multimodal Multitask Multilingual Learning](https://mlsanity.com/paper/2303.12489/) (2023)
- [Early Classifying Multimodal Sequences](https://mlsanity.com/paper/2305.01151/) (2023)
- [S-Omninet: Structured Data Enhanced Universal Multimodal Learning Architecture](https://mlsanity.com/paper/2307.00226/) (2023)
- [Dynamic Inter-Class Confusion-Aware Encoder for Audio-Visual Fusion in Human Activity Recognition](https://mlsanity.com/paper/2507.09323/) (2025)

## Abstract (arXiv)

Transformer is a popularly used neural network architecture, especially for language understanding. We introduce an extended and unified architecture that can be used for tasks involving a variety of modalities like image, text, videos, etc. We propose a spatio-temporal cache mechanism that enables learning spatial dimension of the input in addition to the hidden states corresponding to the temporal input sequence. The proposed architecture further enables a single model to support tasks with multiple input modalities as well as asynchronous multi-task learning, thus we refer to it as OmniNet. For example, a single instance of OmniNet can concurrently learn to perform the tasks of part-of-speech tagging, image captioning, visual question answering and video activity recognition. We demonstrate that training these four tasks together results in about three times compressed model while retaining the performance in comparison to training them individually. We also show that using this neural network pre-trained on some modalities assists in learning unseen tasks such as video captioning and video question answering. This illustrates the generalization capacity of the self-attention mechanism on the spatio-temporal cache present in OmniNet.