mlsanityData to 14 Oct 2026

Task-Adaptive Tokenization: Enhancing Long-Form Text Generation Efficacy in Mental Health and Beyond

arXiv 2310.05317 ↗First posted 9 Oct 2023cs.CL cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

We propose task-adaptive tokenization as a way to adapt the generation pipeline to the specifics of a downstream task and enhance long-form generation in mental health. Inspired by insights from cognitive science, our task-adaptive tokenizer samples variable segmentations from multiple outcomes, with sampling probabilities optimized based on task-specific data. We introduce a strategy for building a specialized vocabulary and introduce a vocabulary merging protocol that allows for the integration of task-specific tokens into the pre-trained model's tokenization step. Through extensive experiments on psychological question-answering tasks in both Chinese and English, we find that our task-adaptive tokenization approach brings a significant improvement in generation performance while using up to 60% fewer tokens. Preliminary experiments point to promising results when using our tokenization approach with very large language models.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 35 of its 71 references that are in the map. First posted on arXiv 9 Oct 2023.

Builds on

35 of 71 references.

References · 35

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  2. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2020Longformer: The Long-Document Transformercited by 3,413
  5. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  7. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  8. 2020Efficient Transformers: A Surveycited by 790
  9. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  10. 2023Efficient and Effective Text Encoding for Chinese LLaMA and Alpacacited by 305
  11. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  12. 2023Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Modelscited by 192
23 more
  1. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  2. 2021MentalBERT: Publicly Available Pretrained Language Models for Mental Healthcarecited by 161
  3. 2021PsyQA: A Chinese Dataset for Generating Long Counseling Text for Mental Health Supportcited by 92
  4. 2021Is GPT-3 Text Indistinguishable from Human Text? Scarecrow: A Framework for Scrutinizing Machine Textcited by 89
  5. 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
  6. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  7. 2023Tokenization and the Noiseless Channelcited by 59
  8. 2021Efficient Domain Adaptation of Language Models via Adaptive Tokenizationcited by 34
  9. 2020BANG: Bridging Autoregressive and Non-autoregressive Generation with Large Scale Pretrainingcited by 33
  10. 2022EVA2.0: Investigating Open-Domain Chinese Dialogue Systems with Large-Scale Pre-Trainingcited by 31
  11. 2023Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Modelscited by 31
  12. 2021Rare Tokens Degenerate All Tokens: Improving Neural Text Generation via Adaptive Gradient Gating for Rare Token Embeddingscited by 27
  13. 2020CliniQG4QA: Generating Diverse Questions for Domain Adaptation of Clinical Question Answeringcited by 26
  14. 2020Stolen Probability: A Structural Weakness of Neural Language Modelscited by 22
  15. 2022Chatbots for Mental Health Support: Exploring the Impact of Emohaa on Reducing Mental Distress in Chinacited by 17
  16. 2021Bridging Subword Gaps in Pretrain-Finetune Paradigm for Natural Language Generationcited by 14
  17. 2021FastSeq: Make Sequence Generation Fastercited by 13
  18. 2020Expressive Interviewing: A Conversational System for Coping with COVID-19cited by 10
  19. 2021Exploring Self-Identified Counseling Expertise in Online Support Forumscited by 10
  20. 2022ELMER: A Non-Autoregressive Pre-trained Language Model for Efficient and Effective Text Generationcited by 10
  21. 2022Time-aware Prompting for Text Generationcited by 10
  22. 2022Training a Tokenizer for Free with Private Federated Learningcited by 4
  23. 2023Now It Sounds Like You: Learning Personalized Vocabulary On Devicecited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 10; 2026: 4.

23 papers.

Sort:

By area:

  1. 2024Large Language Models in Mental Health Care: a Scoping Reviewcited by 34
  2. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  3. 2025AdaptiVocab: Enhancing LLM Efficiency in Focused Domains through Lightweight Vocabulary Adaptationcited by 15
  4. 2025A Cognitive Writing Perspective for Constrained Long-Form Text Generationcited by 14
  5. 2024Gold Panning in Vocabulary: An Adaptive Method for Vocabulary Expansion of Domain-Specific LLMscited by 12
  6. 2024Knowledge Planning in Large Language Models for Domain-Aligned Counseling Summarizationcited by 11
  7. 2024TransMI: A Framework to Create Strong Baselines from Multilingual Pretrained Language Models for Transliterated Datacited by 9
  8. 2025TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarkingcited by 9
  9. 2024MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarizationcited by 8
  10. 2025FLEXITOKENS: Flexible Tokenization for Evolving Language Modelscited by 8
  11. 2024"I Like Sunnie More Than I Expected!": Exploring User Expectation and Perception of an Anthropomorphic LLM-based Conversational Agent for Well-Being Supportcited by 5
  12. 2025zip2zip: Inference-Time Adaptive Tokenization via Online Compressioncited by 5
11 more
  1. 2025SuperWriter: Reflection-Driven Long-Form Generation with Large Language Modelscited by 5
  2. 2024Chumor 2.0: Towards Benchmarking Chinese Humor Understandingcited by 4
  3. 2025Evaluation of LLMs in Medical Text Summarization: The Role of Vocabulary Adaptation in High OOV Settingscited by 4
  4. 2024Chumor 1.0: A Truly Funny and Challenging Chinese Humor Understanding Dataset from Ruo Zhi Bacited by 3
  5. 2025HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initializationcited by 3
  6. 2025CoVE: Compressed Vocabulary Expansion Makes Better LLM-based Recommender Systemscited by 3
  7. 2026AdaptBPE: From General Purpose to Specialized Tokenizerscited by 3
  8. 2026Responsible Evaluation of AI for Mental Healthcited by 2
  9. 2025Tokenization Multiplicity Leads to Arbitrary Price Variation in LLM-as-a-servicecited by 0
  10. 2026Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarizationcited by 0
  11. 2026zip2zip++: Towards Inference-Time Token Compression with Minimal Quality Losscited by 0