mlsanityData to 14 Oct 2026

UPRPRC: Unified Pipeline for Reproducing Parallel Resources -- Corpus from the United Nations

arXiv 2509.15789 ↗First posted 19 Sep 2025cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The quality and accessibility of multilingual datasets are crucial for advancing machine translation. However, previous corpora built from United Nations documents have suffered from issues such as opaque process, difficulty of reproduction, and limited scale. To address these challenges, we introduce a complete end-to-end solution, from data acquisition via web scraping to text alignment. The entire process is fully reproducible, with a minimalist single-machine example and optional distributed computing steps for scalability. At its core, we propose a new Graph-Aided Paragraph Alignment (GAPA) algorithm for efficient and flexible paragraph-level alignment. The resulting corpus contains over 713 million English tokens, more than doubling the scale of prior work. To the best of our knowledge, this represents the largest publicly available parallel corpus composed entirely of human-translated, non-AI-generated content. Our code and corpus are accessible under the MIT License.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 8 of its 24 references that are in the map. First posted on arXiv 19 Sep 2025.

Builds on

8 of 24 references.

References · 8

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023A Survey of Large Language Modelscited by 2,576
  3. 2024Qwen2 Technical Reportcited by 2,120
  4. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  5. 2023A Comprehensive Overview of Large Language Modelscited by 648
  6. 2025From Unaligned to Aligned: Scaling Multilingual LLMs with Multi-Way Parallel Corporacited by 7
  7. 2021Graph Algorithms for Multiparallel Word Alignmentcited by 4
  8. 2022Graph Neural Networks for Multiparallel Word Alignmentcited by 4

Built on by

Not cited yet.