# From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

arXiv: https://arxiv.org/abs/2411.15577 · HTML: https://mlsanity.com/paper/2411.15577/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 13 of its 48 references that are in the map.
- First posted on arXiv 23 Nov 2024.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 13 of 13)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Neural Machine Translation with Byte-Level Subwords](https://mlsanity.com/paper/1909.03341/) (2019)
- [NV-Retriever: Improving text embedding models with effective hard-negative mining](https://mlsanity.com/paper/2407.15831/) (2024)
- [A Benchmark for Learning to Translate a New Language from One Grammar Book](https://mlsanity.com/paper/2309.16575/) (2023)
- [Building Machine Translation Systems for the Next Thousand Languages](https://mlsanity.com/paper/2205.03983/) (2022)
- [Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptions](https://mlsanity.com/paper/2402.18025/) (2024)
- [The Past, Present, and Future of Typological Databases in NLP](https://mlsanity.com/paper/2310.13440/) (2023)
- [Low Resourced Machine Translation via Morpho-syntactic Modeling: The Case of Dialectal Arabic](https://mlsanity.com/paper/1712.06273/) (2017)

## Built on by (most cited first, 2 of 2)

- [Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languages](https://mlsanity.com/paper/2502.09778/) (2025)
- [Testing the Limits of Machine Translation from One Book](https://mlsanity.com/paper/2508.06665/) (2025)

## Abstract (arXiv)

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at urlhttps://github.com/al-the-eigenvalue/RAG-on-grammars.