mlsanityData to 14 Oct 2026

Benchmarking Procedural Language Understanding for Low-Resource Languages: A Case Study on Turkish

arXiv 2309.06698 ↗First posted 13 Sep 2023cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

Understanding procedural natural language (e.g., step-by-step instructions) is a crucial step to execution and planning. However, while there are ample corpora and downstream tasks available in English, the field lacks such resources for most languages. To address this gap, we conduct a case study on Turkish procedural texts. We first expand the number of tutorials in Turkish wikiHow from 2,000 to 52,000 using automated translation tools, where the translation quality and loyalty to the original meaning are validated by a team of experts on a random set. Then, we generate several downstream tasks on the corpus, such as linking actions, goal inference, and summarization. To tackle these tasks, we implement strong baseline models via fine-tuning large language-specific models such as TR-BART and BERTurk, as well as multilingual models such as mBART, mT5, and XLM. We find that language-specific models consistently outperform their multilingual models by a significant margin across most procedural language understanding (PLU) tasks. We release our corpus, downstream tasks and the baseline models with https://github.com/ GGLAB-KU/turkish-plu.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 35 of its 61 references that are in the map. First posted on arXiv 13 Sep 2023.

Builds on

35 of 61 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  8. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  9. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  10. 2017Billion-scale similarity search with GPUscited by 2,497
  11. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  12. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
23 more
  1. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  3. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,033
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 1,001
  6. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  7. 2020Language-agnostic BERT Sentence Embeddingcited by 627
  8. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 593
  9. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  10. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
  11. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  12. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  13. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  14. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  15. 2020Reasoning about Goals, Steps, and Temporal Ordering with WikiHowcited by 66
  16. 2021Visual Goal-Step Inference using wikiHowcited by 43
  17. 2020Analogous Process Structure Induction for Sub-event Sequence Predictioncited by 26
  18. 2022Show Me More Details: Discovering Hierarchies of Procedures from Semi-structured Web Datacited by 22
  19. 2019Learning Household Task Knowledge from WikiHow Descriptionscited by 15
  20. 2020A Recipe for Creating Multimodal Aligned Datasets for Sequential Taskscited by 15
  21. 2017Sequence to Sequence Learning for Event Predictioncited by 9
  22. 2022Mukayese: Turkish NLP Strikes Backcited by 9
  23. 2020Why Not Simply Translate? A First Swedish Evaluation Benchmark for Semantic Similaritycited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 2; 2026: 1.

5 papers.

  1. 2025Evaluating the Quality of Benchmark Datasets for Low-Resource Languages: A Case Study on Turkishcited by 13
  2. 2025Cetvel: A Unified Benchmark for Evaluating Language Understanding, Generation and Cultural Capacity of LLMs for Turkishcited by 8
  3. 2026TurkBench: A Benchmark for Evaluating Turkish Large Language Modelscited by 5
  4. 2024WorldAPIs: The World Is Worth How Many APIs? A Thought Experimentcited by 4
  5. 2024KoDialogBench: Evaluating Conversational Understanding of Language Models with Korean Dialogue Benchmarkcited by 2