mlsanityData to 14 Oct 2026

In-Place Indic Tokenizer Expansion and Continual Pretraining of Nemotron-3-Nano-30B-A3B

arXiv 2610.15373 ↗First posted 11 Oct 2026cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

We adapted Nemotron-3-Nano-30B-A3B for eight Indian languages without changing its backbone architecture. We expanded the vocabulary in place from 131,072 to 163,072 entries using continued byte-pair encoding and AdaptMix rebalancing, reducing tokens per word by 38.6%. We initialized new input and output rows by averaging their original constituent rows, then trained them on 90% Indic and 10% English text before updating the backbone. Continual pretraining then progressed through broad training, high-quality annealing, and long-context training over approximately 1.27 trillion tokens. The annealed checkpoint gained 12.9 points on the Indic MILU benchmark but lost 3.4 on English MMLU. We also report practical negative results. Direct token addition underperformed continued byte-pair encoding. AdamW weight decay changed rows intended to stay fixed. Perplexity on an Indic-heavy holdout did not select the learning rate that best preserved English performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 30 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

24 of 30 references.

References · 24

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  11. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 861
  12. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
12 more
  1. 2023The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variantscited by 311
  2. 2024Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluationcited by 191
  3. 2021Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languagescited by 136
  4. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  5. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  6. 2024IndicGenBench: A Multilingual Benchmark to Evaluate Generation Capabilities of LLMs on Indic Languagescited by 56
  7. 2024MILU: A Multi-task Indic Language Understanding Benchmarkcited by 27
  8. 2025SANSKRITI: A Comprehensive Benchmark for Evaluating Language Models' Knowledge of Indian Culturecited by 16
  9. 2025IndicMMLU-Pro: Benchmarking Indic Large Language Models on Multi-Task Language Understandingcited by 10
  10. 2025The Art of Breaking Words: Rethinking Multilingual Tokenizer Designcited by 6
  11. 2025IndicParam: Benchmark to evaluate LLMs on low-resource Indic Languagescited by 3
  12. 2026In-Place Tokenizer Expansion for Pre-trained LLMscited by 1

Built on by

Not cited yet.