{"list":"other","total":66,"rows":[{"id":"1810.04805","title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding","created":"2018-10-11","cited_by_count":40399},{"id":"2005.14165","title":"Language Models are Few-Shot Learners","created":"2020-05-28","cited_by_count":32419},{"id":"1508.07909","title":"Neural Machine Translation of Rare Words with Subword Units","created":"2015-08-31","cited_by_count":2918},{"id":"1808.06226","title":"SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing","created":"2018-08-19","cited_by_count":1989},{"id":"1804.10959","title":"Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates","created":"2018-04-29","cited_by_count":639},{"id":"2105.13626","title":"ByT5: Towards a token-free future with pre-trained byte-to-byte models","created":"2021-05-28","cited_by_count":509},{"id":"2012.15613","title":"How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models","created":"2020-12-31","cited_by_count":318},{"id":"2305.15425","title":"Language Model Tokenizers Introduce Unfairness Between Languages","created":"2023-05-17","cited_by_count":214},{"id":"2004.03720","title":"Byte Pair Encoding is Suboptimal for Language Model Pretraining","created":"2020-04-07","cited_by_count":192},{"id":"2305.13707","title":"Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models","created":"2023-05-23","cited_by_count":192},{"id":"2106.12672","title":"Charformer: Fast Character Transformers via Gradient-based Subword Tokenization","created":"2021-06-23","cited_by_count":160},{"id":"2305.18486","title":"A Systematic Study and Comprehensive Evaluation of ChatGPT on Benchmark Datasets","created":"2023-05-29","cited_by_count":123},{"id":"2407.13623","title":"Scaling Laws with Vocabulary: Larger Models Deserve Larger Vocabularies","created":"2024-07-18","cited_by_count":118},{"id":"2012.15562","title":"UNKs Everywhere: Adapting Multilingual Language Models to New Scripts","created":"2020-12-31","cited_by_count":113},{"id":"2310.08754","title":"Tokenizer Choice For LLM Training: Negligible or Crucial?","created":"2023-10-12","cited_by_count":108},{"id":"2312.00738","title":"SeaLLMs -- Large Language Models for Southeast Asia","created":"2023-12-01","cited_by_count":104},{"id":"2401.01055","title":"LLaMA Beyond English: An Empirical Study on Language Capability Transfer","created":"2024-01-02","cited_by_count":95},{"id":"2401.06059","title":"Investigating Data Contamination for Pre-training Language Models","created":"2024-01-11","cited_by_count":91},{"id":"2402.18376","title":"Tokenization Is More Than Compression","created":"2024-02-28","cited_by_count":89},{"id":"2407.04069","title":"A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations","created":"2024-07-04","cited_by_count":85},{"id":"2507.07955","title":"Dynamic Chunking for End-to-End Hierarchical Sequence Modeling","created":"2025-07-10","cited_by_count":72},{"id":"2503.13423","title":"SuperBPE: Space Travel for Language Models","created":"2025-03-17","cited_by_count":63},{"id":"2204.08832","title":"Impact of Tokenization on Language Models: An Analysis for Turkish","created":"2022-04-19","cited_by_count":62},{"id":"2411.14198","title":"Why do language models perform worse for morphologically complex languages?","created":"2024-11-21","cited_by_count":62},{"id":"2403.06265","title":"Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance","created":"2024-03-10","cited_by_count":61},{"id":"2501.16975","title":"Over-Tokenized Transformer: Vocabulary is Generally Worth Scaling","created":"2025-01-28","cited_by_count":44},{"id":"2506.12103","title":"The Amazon Nova Family of Models: Technical Report and Model Card","created":"2025-03-17","cited_by_count":41},{"id":"2106.13474","title":"Adapt-and-Distill: Developing Small, Fast and Effective Pretrained Language Models for Domains","created":"2021-06-25","cited_by_count":36},{"id":"2306.16837","title":"A Formal Perspective on Byte-Pair Encoding","created":"2023-06-29","cited_by_count":36},{"id":"2310.19233","title":"Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective","created":"2023-10-30","cited_by_count":34},{"id":"2403.01289","title":"Greed is All You Need: An Evaluation of Tokenizer Inference Methods","created":"2024-03-02","cited_by_count":34},{"id":"2402.00841","title":"Tiny Titans: Can Smaller Large Language Models Punch Above Their Weight in the Real World for Meeting Summarization?","created":"2024-02-01","cited_by_count":33},{"id":"2405.17067","title":"Tokenization Matters! Degrading Large Language Models through Challenging Their Tokenization","created":"2024-05-27","cited_by_count":32},{"id":"2504.00178","title":"Boundless Byte Pair Encoding: Breaking the Pre-tokenization Barrier","created":"2025-03-31","cited_by_count":32},{"id":"2407.08818","title":"MAGNET: Improving the Multilingual Fairness of Language Models with Adaptive Gradient-Based Tokenization","created":"2024-07-11","cited_by_count":31},{"id":"2409.04599","title":"BPE Gets Picky: Efficient Vocabulary Refinement During Tokenizer Training","created":"2024-09-06","cited_by_count":31},{"id":"2310.13348","title":"Analyzing Cognitive Plausibility of Subword Tokenization","created":"2023-10-20","cited_by_count":30},{"id":"1909.12440","title":"Improving Pre-Trained Multilingual Models with Vocabulary Expansion","created":"2019-09-26","cited_by_count":28},{"id":"2502.00894","title":"MorphBPE: A Morpho-Aware Tokenizer Bridging Linguistic Complexity for Efficient LLM Training Across Morphologies","created":"2025-02-02","cited_by_count":25},{"id":"2406.14670","title":"Exploring Design Choices for Building Language-Specific LLMs","created":"2024-06-20","cited_by_count":22},{"id":"2501.10322","title":"Hierarchical Autoregressive Transformers: Combining Byte- and Word-Level Processing for Robust, Adaptable Language Models","created":"2025-01-17","cited_by_count":22},{"id":"2507.06378","title":"Evaluating Morphological Alignment of Tokenizers in 70 Languages","created":"2025-07-08","cited_by_count":22},{"id":"2409.11501","title":"Egalitarian Language Representation in Language Models: It All Begins with Tokenizers","created":"2024-09-17","cited_by_count":20},{"id":"2404.17808","title":"Scaffold-BPE: Enhancing Byte Pair Encoding for Large Language Models with Simple and Effective Scaffold Token Removal","created":"2024-04-27","cited_by_count":16},{"id":"2411.18553","title":"Retrofitting Large Language Models with Dynamic Tokenization","created":"2024-11-27","cited_by_count":16},{"id":"2105.12410","title":"Joint Optimization of Tokenization and Downstream Model","created":"2021-05-26","cited_by_count":15},{"id":"2502.20273","title":"How Much is Enough? The Diminishing Returns of Tokenization Training Data","created":"2025-02-27","cited_by_count":15},{"id":"1810.08641","title":"Optimizing Segmentation Granularity for Neural Machine Translation","created":"2018-10-19","cited_by_count":14},{"id":"2406.13560","title":"Lexically Grounded Subword Segmentation","created":"2024-06-19","cited_by_count":13},{"id":"2502.15343","title":"Tokenization is Sensitive to Language Variation","created":"2025-02-21","cited_by_count":13},{"id":"2410.01188","title":"Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMs","created":"2024-10-02","cited_by_count":12},{"id":"2306.17649","title":"Biomedical Language Models are Robust to Sub-optimal Tokenization","created":"2023-06-30","cited_by_count":9},{"id":"2406.03776","title":"XL-HeadTags: Leveraging Multimodal Retrieval Augmentation for the Multilingual Generation of News Headlines and Tags","created":"2024-06-06","cited_by_count":9},{"id":"2210.06525","title":"Subword Segmental Language Modelling for Nguni Languages","created":"2022-10-12","cited_by_count":8},{"id":"2410.03884","title":"KidLM: Advancing Language Models for Children -- Early Insights and Future Directions","created":"2024-10-04","cited_by_count":8},{"id":"2412.06926","title":"When Every Token Counts: Optimal Segmentation for Low-Resource Language Models","created":"2024-12-09","cited_by_count":8},{"id":"2410.23684","title":"Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers","created":"2024-10-31","cited_by_count":7},{"id":"2502.12560","title":"How does a Language-Specific Tokenizer affect LLMs?","created":"2025-02-18","cited_by_count":6},{"id":"2306.01393","title":"Assessing the Importance of Frequency versus Compositionality for Subword-based Tokenization in NMT","created":"2023-06-02","cited_by_count":5},{"id":"2410.18836","title":"From English-Centric to Effective Bilingual: LLMs with Custom Tokenizers for Underrepresented Languages","created":"2024-10-24","cited_by_count":5},{"id":"2412.12841","title":"Benchmarking and Understanding Compositional Relational Reasoning of LLMs","created":"2024-12-17","cited_by_count":4},{"id":"2503.17247","title":"KL3M Tokenizers: A Family of Domain-Specific and Character-Level Tokenizers for Legal, Financial, and Preprocessing Applications","created":"2025-03-21","cited_by_count":4},{"id":"2509.00285","title":"OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews","created":"2025-08-30","cited_by_count":4},{"id":"2512.20757","title":"TokSuite: Measuring the Impact of Tokenizer Choice on Language Model Behavior","created":"2025-12-23","cited_by_count":4},{"id":"2506.07541","title":"Bit-level BPE: Below the byte boundary","created":"2025-06-09","cited_by_count":3},{"id":"2510.09947","title":"Beyond Fertility: Analyzing STRR as a Metric for Multilingual Tokenization Evaluation","created":"2025-10-11","cited_by_count":3}],"next_cursor":null}