mlsanityData to 14 Oct 2026

SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing

arXiv 1808.06226 ↗First posted 19 Aug 2018cs.CLcited by 1,989

Not on the map. Why?

Cited by 41 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper describes SentencePiece, a language-independent subword tokenizer and detokenizer designed for Neural-based text processing, including Neural Machine Translation. It provides open-source C++ and Python implementations for subword units. While existing subword segmentation tools assume that the input is pre-tokenized into word sequences, SentencePiece can train subword models directly from raw sentences, which allows us to make a purely end-to-end and language independent system. We perform a validation experiment of NMT on English-Japanese machine translation, and find that it is possible to achieve comparable accuracy to direct subword training from raw sentences. We also compare the performance of subword training and segmentation with various configurations. SentencePiece is available under the Apache 2 license at https://github.com/google/sentencepiece.

Key facts. Cited by 1,989 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2018: 1; 2019: 0; 2020: 2; 2021: 322; 2022: 350; 2023: 421; 2024: 441; 2025: 245; 2026: 207.

1,989 papers.

Sort:

By area:

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  8. 2021Program Synthesis with Large Language Modelscited by 3,637
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,300
  10. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2023A Survey of Large Language Modelscited by 2,576
38 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
  4. 2025Gemma 3 Technical Reportcited by 1,824
  5. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  6. 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
  7. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  8. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  9. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  10. 2022LaMDA: Language Models for Dialog Applicationscited by 1,220
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  12. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,202
  13. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  14. 2023BloombergGPT: A Large Language Model for Financecited by 939
  15. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  16. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  17. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  18. 2022A Generalist Agentcited by 808
  19. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 740
  20. 2024Yi: Open Foundation Models by 01.AIcited by 719
  21. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  22. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  23. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  24. 2023Baichuan 2: Open Large-scale Language Modelscited by 670
  25. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  26. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  27. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 629
  28. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  29. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 568
  30. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  31. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  32. 2023MotionGPT: Human Motion as a Foreign Languagecited by 527
  33. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 520
  34. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  35. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  36. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
  37. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  38. 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419

50 most cited of 1,989 All 1,989 (JSON)