# Enhancing Pre-trained Chinese Character Representation with Word-aligned Attention

arXiv: https://arxiv.org/abs/1911.02821 · HTML: https://mlsanity.com/paper/1911.02821/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 7 Nov 2019.

## Built on by (most cited first, 7 of 7)

- [Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey](https://mlsanity.com/paper/2302.10035/) (2023)
- [ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders](https://mlsanity.com/paper/2105.01279/) (2021)
- [Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://mlsanity.com/paper/2104.07204/) (2021)
- [LET: Linguistic Knowledge Enhanced Graph Transformer for Chinese Short Text Matching](https://mlsanity.com/paper/2102.12671/) (2021)
- [A Survey of Knowledge Enhanced Pre-trained Models](https://mlsanity.com/paper/2110.00269/) (2021)
- [elBERto: Self-supervised Commonsense Learning for Question Answering](https://mlsanity.com/paper/2203.09424/) (2022)
- [Constructing Word-Context-Coupled Space Aligned with Associative Knowledge Relations for Interpretable Language Modeling](https://mlsanity.com/paper/2305.11543/) (2023)

## Abstract (arXiv)

Most Chinese pre-trained models take character as the basic unit and learn representation according to character's external contexts, ignoring the semantics expressed in the word, which is the smallest meaningful utterance in Chinese. Hence, we propose a novel word-aligned attention to exploit explicit word information, which is complementary to various character-based Chinese pre-trained language models. Specifically, we devise a pooling mechanism to align the character-level attention to the word level and propose to alleviate the potential issue of segmentation error propagation by multi-source information fusion. As a result, word and character information are explicitly integrated at the fine-tuning procedure. Experimental results on five Chinese NLP benchmark tasks demonstrate that our model could bring another significant gain over several pre-trained models.