Comprehensive Evaluation on Lexical Normalization: Boundary-Aware Approaches for Unsegmented Languages
Not on the map. Why?
Abstract (arXiv)
Lexical normalization research has sought to tackle the challenge of processing informal expressions in user-generated text, yet the absence of comprehensive evaluations leaves it unclear which methods excel across multiple perspectives. Focusing on unsegmented languages, we make three key contributions: (1) creating a large-scale, multi-domain Japanese normalization dataset, (2) developing normalization methods based on state-of-the-art pretrained models, and (3) conducting experiments across multiple evaluation perspectives. Our experiments show that both encoder-only and decoder-only approaches achieve promising results in both accuracy and efficiency.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 22 of its 74 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
22 of 74 references.
References · 22
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
- 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
- 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 554
- 2020The Multilingual Amazon Reviews Corpuscited by 143
10 more
- 2024Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilitiescited by 111
- 2024SeaLLMs 3: Open Foundation and Chat Multilingual Large Language Models for Southeast Asian Languagescited by 36
- 2019A Large-Scale Comparison of Historical Text Normalization Systemscited by 25
- 2025TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Modelscited by 22
- 2021DaN+: Danish Nested Named Entities and Lexical Normalizationcited by 19
- 2023A Chat About Boring Problems: Studying GPT-based text normalizationcited by 10
- 2019Adapting Sequence to Sequence models for Text Normalization in Social Mediacited by 8
- 2021Sequence-to-Sequence Lexical Normalization with Multilingual Transformerscited by 7
- 2021'UFAL at MultiLexNorm 2021: Improving Multilingual Lexical Normalization by Fine-tuning ByT5cited by 7
- 2021User-Generated Text Corpus for Evaluating Japanese Morphological Analysis and Lexical Normalizationcited by 2
Built on by
Citing papers per year: 2025: 1.
1 paper.