mlsanityData to 14 Oct 2026

Leveraging Closed-Access Multilingual Embedding for Automatic Sentence Alignment in Low Resource Languages

arXiv 2311.12179 ↗First posted 20 Nov 2023cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

The importance of qualitative parallel data in machine translation has long been determined but it has always been very difficult to obtain such in sufficient quantity for the majority of world languages, mainly because of the associated cost and also the lack of accessibility to these languages. Despite the potential for obtaining parallel datasets from online articles using automatic approaches, forensic investigations have found a lot of quality-related issues such as misalignment, and wrong language codes. In this work, we present a simple but qualitative parallel sentence aligner that carefully leveraged the closed-access Cohere multilingual embedding, a solution that ranked second in the just concluded #CoHereAIHack 2023 Challenge (see https://ai6lagos.devpost.com). The proposed approach achieved 94.96 and 54.83 f1 scores on FLORES and MAFAND-MT, compared to 3.64 and 0.64 of LASER respectively. Our method also achieved an improvement of more than 5 BLEU scores over LASER, when the resulting datasets were used with MAFAND-MT dataset to train translation models. Our code and data are available for research purposes here (https://github.com/abumafrim/Cohere-Align).

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 16 of its 23 references that are in the map. First posted on arXiv 20 Nov 2023.

Builds on

16 of 23 references.

References · 16

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
  5. 2020Beyond English-Centric Multilingual Machine Translationcited by 561
  6. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  7. 2017Word Translation Without Parallel Datacited by 487
  8. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  9. 2021Quality at a Glance: An Audit of Web-Crawled Multilingual Datasetscited by 243
  10. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  11. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  12. 2022A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translationcited by 98
4 more
  1. 2022Bitext Mining Using Distilled Sentence Representations for Low-Resource Languagescited by 48
  2. 2018A Corpus for Multilingual Document Classification in Eight Languagescited by 42
  3. 2022Separating Grains from the Chaff: Using Data Filtering to Improve Multilingual Translation for Low-Resourced African Languagescited by 6
  4. 2020A Hybrid Approach for Improved Low Resource Neural Machine Translation using Monolingual Datacited by 3

Built on by

Not cited yet.