# MLM: A Benchmark Dataset for Multitask Learning with Multiple Languages and Modalities

arXiv: https://arxiv.org/abs/2008.06376 · HTML: https://mlsanity.com/paper/2008.06376/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Aug 2020.

## Built on by (most cited first, 10 of 10)

- [WIT: Wikipedia-based Image Text Dataset for Multimodal Multilingual Machine Learning](https://mlsanity.com/paper/2103.01913/) (2021)
- [IGLUE: A Benchmark for Transfer Learning across Modalities, Tasks, and Languages](https://mlsanity.com/paper/2201.11732/) (2022)
- [Conversational Question Answering over Knowledge Graphs with Transformer and Graph Attention Networks](https://mlsanity.com/paper/2104.01569/) (2021)
- [SCoFT: Self-Contrastive Fine-Tuning for Equitable Image Generation](https://mlsanity.com/paper/2401.08053/) (2024)
- [ArtELingo: A Million Emotion Annotations of WikiArt with Emphasis on Diversity over Language and Culture](https://mlsanity.com/paper/2211.10780/) (2022)
- [A Priority Map for Vision-and-Language Navigation with Trajectory Plans and Feature-Location Cues](https://mlsanity.com/paper/2207.11717/) (2022)
- [VOGUE: Answer Verbalization through Multi-Task Learning](https://mlsanity.com/paper/2106.13316/) (2021)
- [OEKG: The Open Event Knowledge Graph](https://mlsanity.com/paper/2302.14688/) (2023)
- [Legal-HNet: Mixing Legal Long-Context Tokens with Hartley Transform](https://mlsanity.com/paper/2311.05089/) (2023)
- [VISAT: Benchmarking Adversarial and Distribution Shift Robustness in Traffic Sign Recognition with Visual Attributes](https://mlsanity.com/paper/2510.26833/) (2025)

## Abstract (arXiv)

In this paper, we introduce the MLM (Multiple Languages and Modalities) dataset - a new resource to train and evaluate multitask systems on samples in multiple modalities and three languages. The generation process and inclusion of semantic data provide a resource that further tests the ability for multitask systems to learn relationships between entities. The dataset is designed for researchers and developers who build applications that perform multiple tasks on data encountered on the web and in digital archives. A second version of MLM provides a geo-representative subset of the data with weighted samples for countries of the European Union. We demonstrate the value of the resource in developing novel applications in the digital humanities with a motivating use case and specify a benchmark set of tasks to retrieve modalities and locate entities in the dataset. Evaluation of baseline multitask and single task systems on the full and geo-representative versions of MLM demonstrate the challenges of generalising on diverse data. In addition to the digital humanities, we expect the resource to contribute to research in multimodal representation learning, location estimation, and scene understanding.