# SynDARin: Synthesising Datasets for Automated Reasoning in Low-Resource Languages

arXiv: https://arxiv.org/abs/2406.14425 · HTML: https://mlsanity.com/paper/2406.14425/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 20 of its 35 references that are in the map.
- First posted on arXiv 20 Jun 2024.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)
- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [BERTopic: Neural topic modeling with a class-based TF-IDF procedure](https://mlsanity.com/paper/2203.05794/) (2022)
- [Reading Wikipedia to Answer Open-Domain Questions](https://mlsanity.com/paper/1704.00051/) (2017)
- [MLQA: Evaluating Cross-lingual Extractive Question Answering](https://mlsanity.com/paper/1910.07475/) (2019)
- [Hypothesis Only Baselines in Natural Language Inference](https://mlsanity.com/paper/1805.01042/) (2018)
- [MEGA: Multilingual Evaluation of Generative AI](https://mlsanity.com/paper/2303.12528/) (2023)
- [Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations](https://mlsanity.com/paper/2310.07849/) (2023)
- [Translation Artifacts in Cross-lingual Transfer Learning](https://mlsanity.com/paper/2004.04721/) (2020)
- [Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering](https://mlsanity.com/paper/2010.12643/) (2020)
- [Automatic Spanish Translation of the SQuAD Dataset for Multilingual Question Answering](https://mlsanity.com/paper/1912.05200/) (2019)
- [From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers](https://mlsanity.com/paper/2005.00633/) (2020)
- [Multilingual Extractive Reading Comprehension by Runtime Machine Translation](https://mlsanity.com/paper/1809.03275/) (2018)
- [Towards Zero-Shot Multilingual Synthetic Question and Answer Generation for Cross-Lingual Reading Comprehension](https://mlsanity.com/paper/2010.12008/) (2020)

## Built on by (most cited first, 1 of 1)

- [ADAGE: A Language-Agnostic Pipeline for Analogical Reasoning Evaluation](https://mlsanity.com/paper/2607.23058/) (2026)

## Abstract (arXiv)

Question Answering (QA) datasets have been instrumental in developing and evaluating Large Language Model (LLM) capabilities. However, such datasets are scarce for languages other than English due to the cost and difficulties of collection and manual annotation. This means that producing novel models and measuring the performance of multilingual LLMs in low-resource languages is challenging. To mitigate this, we propose SynDARin, a method for generating and validating QA datasets for low-resource languages. We utilize parallel content mining to obtain textithuman-curated paragraphs between English and the target language. We use the English data as context to textitgenerate synthetic multiple-choice (MC) question-answer pairs, which are automatically translated and further validated for quality. Combining these with their designated non-English textithuman-curated paragraphs form the final QA dataset. The method allows to maintain the content quality, reduces the likelihood of factual errors, and circumvents the need for costly annotation. To test the method, we created a QA dataset with 1.2K samples for the Armenian language. The human evaluation shows that 98% of the generated English data maintains quality and diversity in the question types and topics, while the translation validation pipeline can filter out ~70% of data with poor quality. We use the dataset to benchmark state-of-the-art LLMs, showing their inability to achieve human accuracy with some model performances closer to random chance. This shows that the generated dataset is non-trivial and can be used to evaluate reasoning capabilities in low-resource language.