mlsanityData to 13 Oct 2026

BERT Cannot Align Characters

arXiv 2109.09700 ↗First posted 20 Sep 2021cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

In previous work, it has been shown that BERT can adequately align cross-lingual sentences on the word level. Here we investigate whether BERT can also operate as a char-level aligner. The languages examined are English, Fake-English, German and Greek. We show that the closer two languages are, the better BERT can align them on the character level. BERT indeed works well in English to Fake-English alignment, but this does not generalize to natural languages to the same extent. Nevertheless, the proximity of two languages does seem to be a factor. English is more related to German than to Greek and this is reflected in how well BERT aligns them; English to German is better than English to Greek. We examine multiple setups and show that the similarity matrices for natural languages show weaker relations the further apart two languages are.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 18 of its 26 references that are in the map. First posted on arXiv 20 Sep 2021.

Builds on

18 of 26 references.

References · 18

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2016Enriching Word Vectors with Subword Informationcited by 1,581
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,089
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,025
  6. 2019How multilingual is Multilingual BERT?cited by 643
  7. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 577
  8. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  9. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  10. 2020SimAlign: High Quality Word Alignments without Parallel Training Data using Static and Contextualized Embeddingscited by 155
  11. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  12. 2019Jointly Learning to Align and Translate with Transformer Modelscited by 75
6 more
  1. 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
  2. 2020Applying the Transformer to Character-level Transductioncited by 43
  3. 2019Adding Interpretable Attention to Neural Translation Models Improves Word Alignmentcited by 30
  4. 2019Don't Forget the Long Tail! A Comprehensive Analysis of Morphological Generalization in Bilingual Lexicon Inductioncited by 15
  5. 2017From Characters to Words to in Between: Do We Capture Morphology?cited by 13
  6. 2016Neural Network-based Word Alignment through Score Aggregationcited by 3

Built on by

Not cited yet.