# Mimir: Large-scale Multilingual Concept Modeling

arXiv: https://arxiv.org/abs/2605.25263 · HTML: https://mlsanity.com/paper/2605.25263/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 24 May 2026.

## Builds on (most cited first, 26 of 26)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyond](https://mlsanity.com/paper/1812.10464/) (2018)
- [XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languages](https://mlsanity.com/paper/2106.13822/) (2021)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [OpenMathInstruct-2: Accelerating AI for Math with Massive Open-Source Instruction Data](https://mlsanity.com/paper/2410.01560/) (2024)
- [FineWeb2: One Pipeline to Scale Them All -- Adapting Pre-Training Data Processing to Every Language](https://mlsanity.com/paper/2506.20920/) (2025)
- [SONAR: Sentence-Level Multimodal and Language-Agnostic Representations](https://mlsanity.com/paper/2308.11466/) (2023)
- [EuroLLM: Multilingual Language Models for Europe](https://mlsanity.com/paper/2409.16235/) (2024)
- [Large Language Models Meet NLP: A Survey](https://mlsanity.com/paper/2405.12819/) (2024)
- [MultiEURLEX -- A multi-lingual and multi-label legal document classification dataset for zero-shot cross-lingual transfer](https://mlsanity.com/paper/2109.00904/) (2021)
- [Large Concept Models: Language Modeling in a Sentence Representation Space](https://mlsanity.com/paper/2412.08821/) (2024)
- [Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation](https://mlsanity.com/paper/2305.15011/) (2023)
- [Multilingual LLMs are Better Cross-lingual In-context Learners with Alignment](https://mlsanity.com/paper/2305.05940/) (2023)
- [LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback](https://mlsanity.com/paper/2406.01771/) (2024)
- [Segment Any Text: A Universal Approach for Robust, Efficient and Adaptable Sentence Segmentation](https://mlsanity.com/paper/2406.16678/) (2024)
- [COPEN: Probing Conceptual Knowledge in Pre-trained Language Models](https://mlsanity.com/paper/2211.04079/) (2022)
- [Where's the Point? Self-Supervised Multilingual Punctuation-Agnostic Sentence Segmentation](https://mlsanity.com/paper/2305.18893/) (2023)
- [Disentangling Language and Culture for Evaluating Multilingual Large Language Models](https://mlsanity.com/paper/2505.24635/) (2025)
- [Grounding Multilingual Multimodal LLMs With Cultural Knowledge](https://mlsanity.com/paper/2508.07414/) (2025)
- [Exploring the Word Sense Disambiguation Capabilities of Large Language Models](https://mlsanity.com/paper/2503.08662/) (2025)
- [Towards Achieving Concept Completeness for Textual Concept Bottleneck Models](https://mlsanity.com/paper/2502.11100/) (2025)
- [When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training](https://mlsanity.com/paper/2601.22851/) (2026)
- [Centurio: On Drivers of Multilingual Ability of Large Vision-Language Model](https://mlsanity.com/paper/2501.05122/) (2025)
- [Beyond Tokens: Concept-Level Training Objectives for LLMs](https://mlsanity.com/paper/2601.11791/) (2026)

## Abstract (arXiv)

Current language modeling approaches are built around tokens. Text corpora are split into tokens, and models are trained by performing computations on these tokens, such as predicting the next token given the preceding ones as context. This paradigm has become the standard in modern language modeling, especially given the outstanding performance obtained by token-based architectures. However, recent works have not only begun to question how language models process and understand meaning from tokens, but also to question whether using higher levels of granularity could advance the research field. This led to the idea of Concept Modeling, that is, to directly train models for next-concept prediction rather than next-token prediction. In this work, we introduce Mimir, a 1.6B Large Concept Model trained for multilingual concept understanding and generation. We leverage a large-scale multilingual pre-training corpus (38,883,987,240 sentences) spanning 46 languages and a large-scale multi-turn, and multilingual instruction-tuning dataset (66,816,428 sentences) covering a total of 35 languages. We extensively evaluate model performance against a language model with a comparable number of parameters.