# BERT Cannot Align Characters

arXiv: https://arxiv.org/abs/2109.09700 · HTML: https://mlsanity.com/paper/2109.09700/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 26 references that are in the map.
- First posted on arXiv 20 Sep 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Cross-lingual Language Model Pretraining](https://mlsanity.com/paper/1901.07291/) (2019)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [How multilingual is Multilingual BERT?](https://mlsanity.com/paper/1906.01502/) (2019)
- [On the Cross-lingual Transferability of Monolingual Representations](https://mlsanity.com/paper/1910.11856/) (2019)
- [Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERT](https://mlsanity.com/paper/1904.09077/) (2019)
- [Cross-Lingual Ability of Multilingual BERT: An Empirical Study](https://mlsanity.com/paper/1912.07840/) (2019)
- [SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddings](https://mlsanity.com/paper/2004.08728/) (2020)
- [CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characters](https://mlsanity.com/paper/2010.10392/) (2020)
- [Jointly Learning to Align and Translate with Transformer Models](https://mlsanity.com/paper/1909.02074/) (2019)
- [CharBERT: Character-aware Pre-trained Language Model](https://mlsanity.com/paper/2011.01513/) (2020)
- [Applying the Transformer to Character-level Transduction](https://mlsanity.com/paper/2005.10213/) (2020)
- [Adding Interpretable Attention to Neural Translation Models Improves Word Alignment](https://mlsanity.com/paper/1901.11359/) (2019)
- [Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Induction](https://mlsanity.com/paper/1909.02855/) (2019)
- [From Characters to Words to in Between: Do We Capture Morphology?](https://mlsanity.com/paper/1704.08352/) (2017)
- [Neural Network-based Word Alignment through Score Aggregation](https://mlsanity.com/paper/1606.09560/) (2016)

## Abstract (arXiv)

In previous work, it has been shown that BERT can adequately align cross-lingual sentences on the word level. Here we investigate whether BERT can also operate as a char-level aligner. The languages examined are English, Fake-English, German and Greek. We show that the closer two languages are, the better BERT can align them on the character level. BERT indeed works well in English to Fake-English alignment, but this does not generalize to natural languages to the same extent. Nevertheless, the proximity of two languages does seem to be a factor. English is more related to German than to Greek and this is reflected in how well BERT aligns them; English to German is better than English to Greek. We examine multiple setups and show that the similarity matrices for natural languages show weaker relations the further apart two languages are.