mlsanityData to 7 Oct 2026

Empirical Analysis of Korean Public AI Hub Parallel Corpora and in-depth Analysis using LIWC

arXiv 2110.15023 ↗First posted 28 Oct 2021cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine translation (MT) system aims to translate source language into target language. Recent studies on MT systems mainly focus on neural machine translation (NMT). One factor that significantly affects the performance of NMT is the availability of high-quality parallel corpora. However, high-quality parallel corpora concerning Korean are relatively scarce compared to those associated with other high-resource languages, such as German or Italian. To address this problem, AI Hub recently released seven types of parallel corpora for Korean. In this study, we conduct an in-depth verification of the quality of corresponding parallel corpora through Linguistic Inquiry and Word Count (LIWC) and several relevant experiments. LIWC is a word-counting software program that can analyze corpora in multiple ways and extract linguistic features as a dictionary base. To the best of our knowledge, this study is the first to use LIWC to analyze parallel corpora in the field of NMT. Our findings suggest the direction of further research toward obtaining the improved quality parallel corpora through our correlation analysis in LIWC and NMT performance.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 68 references that are in the map. First posted on arXiv 28 Oct 2021.

Builds on

21 of 68 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  8. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  9. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  10. 2017Convolutional Sequence to Sequence Learningcited by 550
  11. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  12. 2019Pay Less Attention with Lightweight and Dynamic Convolutionscited by 211
9 more
  1. 2018Assessing Gender Bias in Machine Translation -- A Case Study with Google Translatecited by 121
  2. 2020Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problemcited by 86
  3. 2018On the Impact of Various Types of Noise on Neural Machine Translationcited by 76
  4. 2016Empath: Understanding Topic Signals in Large-Scale Textcited by 51
  5. 2019Using Monolingual Data in Neural Machine Translation: a Systematic Studycited by 21
  6. 2020Parallel Corpus Filtering via Pre-trained Language Modelscited by 15
  7. 2018Not Just Depressed: Bipolar Disorder Prediction on Redditcited by 9
  8. 2021BERT meets LIWC: Exploring State-of-the-Art Language Models for Predicting Communication Behavior in Couples' Conflict Interactionscited by 8
  9. 2018Testing Untestable Neural Machine Translation: An Industrial Casecited by 4

Built on by

Not cited yet.