Stolen Subwords: Importance of Vocabularies for Machine Translation Model Stealing
Not on the map. Why?
Abstract (arXiv)
In learning-based functionality stealing, the attacker is trying to build a local model based on the victim's outputs. The attacker has to make choices regarding the local model's architecture, optimization method and, specifically for NLP models, subword vocabulary, such as BPE. On the machine translation task, we explore (1) whether the choice of the vocabulary plays a role in model stealing scenarios and (2) if it is possible to extract the victim's vocabulary. We find that the vocabulary itself does not have a large effect on the local model's performance. Given gray-box model access, it is possible to collect the victim's vocabulary by collecting the outputs (detokenized subwords on the output). The results of the minimum effect of vocabulary choice are important more broadly for black-box knowledge distillation.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 25 of its 37 references that are in the map. First posted on arXiv 29 Jan 2024.
Builds on
25 of 37 references.
References · 25
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2016Stealing Machine Learning Models via Prediction APIscited by 619
- 2018Knockoff Nets: Stealing Functionality of Black-Box Modelscited by 292
- 2019Facebook FAIR's WMT19 News Translation Task Submissioncited by 179
- 2019Thieves on Sesame Street! Model Extraction of BERT-based APIscited by 139
- 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
- 2019Neural Machine Translation with Byte-Level Subwordscited by 115
- 2020MAZE: Data-Free Model Stealing Attack Using Zeroth-Order Gradient Estimationcited by 99
- 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
13 more
- 2020Imitation Attacks and Defenses for Black-box Machine Translation Systemscited by 81
- 2022Impact of Tokenization on Language Models: An Analysis for Turkishcited by 62
- 2023Tokenization and the Noiseless Channelcited by 59
- 2017Ensemble Distillation for Neural Machine Translationcited by 39
- 2019A Call for Prudent Choice of Subword Merge Operations in Neural Machine Translationcited by 30
- 2021Distilling Holistic Knowledge with Graph Neural Networkscited by 26
- 2019Extraction of Complex DNN Models: Real Threat or Boogeyman?cited by 19
- 2021Student Surpasses Teacher: Imitation Attack for Black-Box NLP APIscited by 18
- 2021MergeDistill: Merging Pre-trained Language Models using Distillationcited by 13
- 2020Distill, Adapt, Distill: Training Small, In-Domain Models for Neural Machine Translationcited by 7
- 2022Knowledge Distillation of Russian Language Models with Reduction of Vocabularycited by 7
- 2021The Highs and Lows of Simple Lexical Domain Adaptation Approaches for Neural Machine Translationcited by 4
- 2021Sampling and Filtering of Neural Machine Translation Distillation Datacited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Survey on Model Extraction Attacks and Defenses for Large Language Models
- 2026Can Watermarking Techniques Help Prevent LLM Model Stealing?
- 2024Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks
- 2024Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models
- 2025Stealix: Model Stealing via Prompt Evolution
- 2025Attack and defense techniques in large language models: A survey and new perspectives
- 2024Purifying Large Language Models by Ensembling a Small Language Model
- 2025Jailbreaking Large Language Models in Infinitely Many Ways