mlsanityData to 7 Oct 2026

Generation with Dynamic Vocabulary

arXiv 2410.08481 ↗First posted 11 Oct 2024cs.CLcited by 5

Not on the map. Why?

Abstract (arXiv)

We introduce a new dynamic vocabulary for language models. It can involve arbitrary text spans during generation. These text spans act as basic generation bricks, akin to tokens in the traditional static vocabularies. We show that, the ability to generate multi-tokens atomically improve both generation quality and efficiency (compared to the standard language model, the MAUVE metric is increased by 25%, the latency is decreased by 20%). The dynamic vocabulary can be deployed in a plug-and-play way, thus is attractive for various downstream applications. For example, we demonstrate that dynamic vocabulary can be applied to different domains in a training-free manner. It also helps to generate reliable citations in question answering tasks (substantially enhancing citation results without compromising answer accuracy).

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

44 of 51 references.

References · 44

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  10. 2017Billion-scale similarity search with GPUscited by 2,477
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
32 more
  1. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  2. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  3. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  4. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2021Ethical and social risks of harm from Language Modelscited by 957
  8. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  9. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  10. 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 719
  11. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
  12. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  13. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  14. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  15. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  16. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  17. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  18. 2023Learning to Compress Prompts with Gist Tokenscited by 379
  19. 2023Adapting Language Models to Compress Contextscited by 350
  20. 2017Six Challenges for Neural Machine Translationcited by 334
  21. 2022Teaching language models to support answers with verified quotescited by 290
  22. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  23. 2022Recurrent Memory Transformercited by 197
  24. 2018Online normalizer calculation for softmaxcited by 154
  25. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  26. 2019Neural Machine Translation: A Review and Surveycited by 99
  27. 2021Efficient Nearest Neighbor Language Modelscited by 93
  28. 2022Neuro-Symbolic Language Modeling with Automaton-augmented Retrievalcited by 60
  29. 2022Handling and Presenting Harmful Text in NLP Researchcited by 43
  30. 2024Fast Vocabulary Transfer for Language Model Compressioncited by 42
  31. 2023Nugget: Neural Agglomerative Embeddings of Textcited by 19
  32. 2024Multi-word Tokenization for Sequence Compressioncited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025TeleMem: Building Long-Term and Multimodal Memory for Agentic AIcited by 14
  2. 2025Protein Design with Dynamic Protein Vocabularycited by 5
  3. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 3
  4. 2026On Discriminative vs. Generative classifiers: Rethinking MLLMs for Action Understandingcited by 1
  5. 2026Flow of Spans: Generalizing Language Models to Dynamic Span-Vocabulary via GFlowNetscited by 0