# Empirical Analysis of Korean Public AI Hub Parallel Corpora and in-depth Analysis using LIWC

arXiv: https://arxiv.org/abs/2110.15023 · HTML: https://mlsanity.com/paper/2110.15023/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 68 references that are in the map.
- First posted on arXiv 28 Oct 2021.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Multilingual Denoising Pre-training for Neural Machine Translation](https://mlsanity.com/paper/2001.08210/) (2020)
- [The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation](https://mlsanity.com/paper/2106.03193/) (2021)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [Pay Less Attention with Lightweight and Dynamic Convolutions](https://mlsanity.com/paper/1901.10430/) (2019)
- [Assessing Gender Bias in Machine Translation -- A Case Study with Google Translate](https://mlsanity.com/paper/1809.02208/) (2018)
- [Reducing Gender Bias in Neural Machine Translation as a Domain Adaptation Problem](https://mlsanity.com/paper/2004.04498/) (2020)
- [On the Impact of Various Types of Noise on Neural Machine Translation](https://mlsanity.com/paper/1805.12282/) (2018)
- [Empath: Understanding Topic Signals in Large-Scale Text](https://mlsanity.com/paper/1602.06979/) (2016)
- [Using Monolingual Data in Neural Machine Translation: a Systematic Study](https://mlsanity.com/paper/1903.11437/) (2019)
- [Parallel Corpus Filtering via Pre-trained Language Models](https://mlsanity.com/paper/2005.06166/) (2020)
- [Not Just Depressed: Bipolar Disorder Prediction on Reddit](https://mlsanity.com/paper/1811.04655/) (2018)
- [BERT meets LIWC: Exploring State-of-the-Art Language Models for Predicting Communication Behavior in Couples' Conflict Interactions](https://mlsanity.com/paper/2106.01536/) (2021)
- [Testing Untestable Neural Machine Translation: An Industrial Case](https://mlsanity.com/paper/1807.02340/) (2018)

## Abstract (arXiv)

Machine translation (MT) system aims to translate source language into target language. Recent studies on MT systems mainly focus on neural machine translation (NMT). One factor that significantly affects the performance of NMT is the availability of high-quality parallel corpora. However, high-quality parallel corpora concerning Korean are relatively scarce compared to those associated with other high-resource languages, such as German or Italian. To address this problem, AI Hub recently released seven types of parallel corpora for Korean. In this study, we conduct an in-depth verification of the quality of corresponding parallel corpora through Linguistic Inquiry and Word Count (LIWC) and several relevant experiments. LIWC is a word-counting software program that can analyze corpora in multiple ways and extract linguistic features as a dictionary base. To the best of our knowledge, this study is the first to use LIWC to analyze parallel corpora in the field of NMT. Our findings suggest the direction of further research toward obtaining the improved quality parallel corpora through our correlation analysis in LIWC and NMT performance.