The boundaries of meaning: a case study in neural machine translation
Not on the map. Why?
Abstract (arXiv)
The success of deep learning in natural language processing raises intriguing questions about the nature of linguistic meaning and ways in which it can be processed by natural and artificial systems. One such question has to do with subword segmentation algorithms widely employed in language modeling, machine translation, and other tasks since 2016. These algorithms often cut words into semantically opaque pieces, such as 'period', 'on', 't', and 'ist' in 'period|on|t|ist'. The system then represents the resulting segments in a dense vector space, which is expected to model grammatical relations among them. This representation may in turn be used to map 'period|on|t|ist' (English) to 'par|od|ont|iste' (French). Thus, instead of being modeled at the lexical level, translation is reformulated more generally as the task of learning the best bilingual mapping between the sequences of subword segments of two languages; and sometimes even between pure character sequences: 'p|e|r|i|o|d|o|n|t|i|s|t' rightarrow 'p|a|r|o|d|o|n|t|i|s|t|e'. Such subword segmentations and alignments are at work in highly efficient end-to-end machine translation systems, despite their allegedly opaque nature. The computational value of such processes is unquestionable. But do they have any linguistic or philosophical plausibility? I attempt to cast light on this question by reviewing the relevant details of the subword segmentation algorithms and by relating them to important philosophical and linguistic debates, in the spirit of making artificial intelligence more transparent and explainable.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 82 references that are in the map. First posted on arXiv 2 Oct 2022.
Builds on
27 of 82 references.
References · 27
- 2017Attention Is All You Needcited by 59,932
- 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
- 2019Compositionality decomposed: how do neural networks generalise?cited by 264
- 2018Colorless green recurrent networks dream hierarchicallycited by 210
- 2020Syntactic Structure from Deep Learningcited by 99
- 2014On Using Very Large Target Vocabulary for Neural Machine Translationcited by 80
- 2019Linguistic generalization and compositionality in modern artificial neural networkscited by 58
15 more
- 2021The paradox of the compositionality of natural language: a neural machine translation case studycited by 58
- 2016Fully Character-Level Neural Machine Translation without Explicit Segmentationcited by 55
- 2014Addressing the Rare Word Problem in Neural Machine Translationcited by 50
- 2016Character-based Neural Machine Translationcited by 33
- 2015Finding Function in Form: Compositional Character Models for Open Vocabulary Word Representationcited by 28
- 2018Revisiting Character-Based Neural Machine Translation with Capacity and Compressioncited by 28
- 2016A Character-Level Decoder without Explicit Segmentation for Neural Machine Translationcited by 21
- 2016Achieving Open Vocabulary Neural Machine Translation with Hybrid Word-Character Modelscited by 18
- 2017From Characters to Words to in Between: Do We Capture Morphology?cited by 13
- 2018Compositional Representation of Morphologically-Rich Input for Neural Machine Translationcited by 12
- 2020Character-Level Translation with Self-attentioncited by 12
- 2014Compositional Morphology for Word Representations and Language Modellingcited by 11
- 2018Learning to Segment Inputs for NMT Favors Character-Level Processingcited by 11
- 2019Character-based NMT with Transformercited by 10
- 2020Character-level Transformer-based Neural Machine Translationcited by 7
Built on by
Citing papers per year: 2023: 1.
1 paper.