mlsanityData to 7 Oct 2026

From MTEB to MTOB: Retrieval-Augmented Classification for Descriptive Grammars

arXiv 2411.15577 ↗First posted 23 Nov 2024cs.CLcited by 2

On the map: Retrieval-augmented generation → Generation and context integration

Abstract (arXiv)

Recent advances in language modeling have demonstrated significant improvements in zero-shot capabilities, including in-context learning, instruction following, and machine translation for extremely under-resourced languages (Tanzer et al., 2024). However, many languages with limited written resources rely primarily on formal descriptions of grammar and vocabulary. In this paper, we introduce a set of benchmarks to evaluate how well models can extract and classify information from the complex descriptions found in linguistic grammars. We present a Retrieval-Augmented Generation (RAG)-based approach that leverages these descriptions for downstream tasks such as machine translation. Our benchmarks encompass linguistic descriptions for 248 languages across 142 language families, focusing on typological features from WALS and Grambank. This set of benchmarks offers the first comprehensive evaluation of language models' in-context ability to accurately interpret and extract linguistic features, providing a critical resource for scaling NLP to low-resource languages. The code and data are publicly available at urlhttps://github.com/al-the-eigenvalue/RAG-on-grammars.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 13 of its 48 references that are in the map. First posted on arXiv 23 Nov 2024.

Builds on

13 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024NV-Retriever: Improving text embedding models with effective hard-negative mining22 Jul
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks22 May

Other references · 11

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 820
  5. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  6. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  7. 2023A Benchmark for Learning to Translate a New Language from One Grammar Bookcited by 74
  8. 2022Building Machine Translation Systems for the Next Thousand Languagescited by 69
  9. 2024Hire a Linguist!: Learning Endangered Languages with In-Context Linguistic Descriptionscited by 12
  10. 2023The Past, Present, and Future of Typological Databases in NLPcited by 7
  11. 2017Low Resourced Machine Translation via Morpho-syntactic Modeling: The Case of Dialectal Arabiccited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Prompt and circumstance: A word-by-word LLM prompting approach to interlinear glossing for low-resource languagescited by 1
  2. 2025Testing the Limits of Machine Translation from One Bookcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Boosting the Capabilities of Compact Models in Low-Data Contexts with Large Language Models and Retrieval-Augmented Generation
  2. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  3. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  4. 2024Contextual Compression in Retrieval-Augmented Generation for Large Language Models: A Survey
  5. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research
  6. 2024RAG and RAU: A Survey on Retrieval-Augmented Language Model in Natural Language Processing
  7. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  8. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models