# AMBERT: A Pre-trained Language Model with Multi-Grained Tokenization

arXiv: https://arxiv.org/abs/2008.11869 · HTML: https://mlsanity.com/paper/2008.11869/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Aug 2020.

## Built on by (most cited first, 22 of 22)

- [CANINE: Pre-training an Efficient Tokenization-Free Encoder for Language Representation](https://mlsanity.com/paper/2103.06874/) (2021)
- [Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLP](https://mlsanity.com/paper/2112.10508/) (2021)
- [CPT: A Pre-Trained Unbalanced Transformer for Both Chinese Language Understanding and Generation](https://mlsanity.com/paper/2109.05729/) (2021)
- [SuperBPE: Space Travel for Language Models](https://mlsanity.com/paper/2503.13423/) (2025)
- [Impact of Tokenization on Language Models: An Analysis for Turkish](https://mlsanity.com/paper/2204.08832/) (2022)
- [Multi-view Subword Regularization](https://mlsanity.com/paper/2103.08490/) (2021)
- [Lattice-BERT: Leveraging Multi-Granularity Representations in Chinese Pre-trained Language Models](https://mlsanity.com/paper/2104.07204/) (2021)
- [ERNIE-mmLayout: Multi-grained MultiModal Transformer for Document Understanding](https://mlsanity.com/paper/2209.08569/) (2022)
- [Toward Building General Foundation Models for Language, Vision, and Vision-Language Understanding Tasks](https://mlsanity.com/paper/2301.05065/) (2023)
- [Sub-Character Tokenization for Chinese Pretrained Language Models](https://mlsanity.com/paper/2106.00400/) (2021)
- [Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Data](https://mlsanity.com/paper/2102.01226/) (2021)
- ["Is Whole Word Masking Always Better for Chinese BERT?": Probing on Chinese Grammatical Error Correction](https://mlsanity.com/paper/2203.00286/) (2022)
- [LICHEE: Improving Language Model Pre-training with Multi-grained Tokenization](https://mlsanity.com/paper/2108.00801/) (2021)
- [MarkBERT: Marking Word Boundaries Improves Chinese BERT](https://mlsanity.com/paper/2203.06378/) (2022)
- [CLOWER: A Pre-trained Language Model with Contrastive Learning over Word and Character Representations](https://mlsanity.com/paper/2208.10844/) (2022)
- [Unsupervised Boundary-Aware Language Model Pretraining for Chinese Sequence Labeling](https://mlsanity.com/paper/2210.15231/) (2022)
- [The Less the Merrier? Investigating Language Representation in Multilingual Models](https://mlsanity.com/paper/2310.13228/) (2023)
- [Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction](https://mlsanity.com/paper/2412.17279/) (2024)
- [Span Fine-tuning for Pre-trained Language Models](https://mlsanity.com/paper/2108.12848/) (2021)
- [FREED++: Improving RL Agents for Fragment-Based Molecule Generation by Thorough Reproduction](https://mlsanity.com/paper/2401.09840/) (2024)
- [Chinese Sequence Labeling with Semi-Supervised Boundary-Aware Language Model Pre-training](https://mlsanity.com/paper/2404.05560/) (2024)
- [Classifier Language Models: Unifying Sparse Finetuning and Adaptive Tokenization for Specialized Classification Tasks](https://mlsanity.com/paper/2508.08635/) (2025)

## Abstract (arXiv)

Pre-trained language models such as BERT have exhibited remarkable performances in many tasks in natural language understanding (NLU). The tokens in the models are usually fine-grained in the sense that for languages like English they are words or sub-words and for languages like Chinese they are characters. In English, for example, there are multi-word expressions which form natural lexical units and thus the use of coarse-grained tokenization also appears to be reasonable. In fact, both fine-grained and coarse-grained tokenizations have advantages and disadvantages for learning of pre-trained language models. In this paper, we propose a novel pre-trained language model, referred to as AMBERT (A Multi-grained BERT), on the basis of both fine-grained and coarse-grained tokenizations. For English, AMBERT takes both the sequence of words (fine-grained tokens) and the sequence of phrases (coarse-grained tokens) as input after tokenization, employs one encoder for processing the sequence of words and the other encoder for processing the sequence of the phrases, utilizes shared parameters between the two encoders, and finally creates a sequence of contextualized representations of the words and a sequence of contextualized representations of the phrases. Experiments have been conducted on benchmark datasets for Chinese and English, including CLUE, GLUE, SQuAD and RACE. The results show that AMBERT can outperform BERT in all cases, particularly the improvements are significant for Chinese. We also develop a method to improve the efficiency of AMBERT in inference, which still performs better than BERT with the same computational cost as BERT.