Inducing Meaningful Units from Character Sequences with Dynamic Capacity Slot Attention
Not on the map. Why?
Abstract (arXiv)
Characters do not convey meaning, but sequences of characters do. We propose an unsupervised distributional method to learn the abstract meaningful units in a sequence of characters. Rather than segmenting the sequence, our Dynamic Capacity Slot Attention model discovers continuous representations of the objects in the sequence, extending an architecture for object discovery in images. We train our model on different languages and evaluate the quality of the obtained representations with forward and reverse probing classifiers. These experiments show that our model succeeds in discovering units which are similar to those proposed previously in form, content and level of abstraction, and which show promise for capturing meaningful information at a higher level of abstraction.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 32 of its 71 references that are in the map. First posted on arXiv 1 Feb 2021.
Builds on
32 of 71 references.
References · 32
- 2017Attention Is All You Needcited by 59,932
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2017Neural Discrete Representation Learningcited by 4,650
- 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2016Pointer Sentinel Mixture Modelscited by 2,992
- 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
- 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
- 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
- 2020Object-Centric Learning with Slot Attentioncited by 865
- 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 632
20 more
- 2015Generating Sentences from a Continuous Spacecited by 607
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
- 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
- 2019Multi-Object Representation Learning with Iterative Variational Inferencecited by 359
- 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
- 2019GENESIS: Generative Scene Inference and Sampling with Object-Centric Latent Representationscited by 227
- 2016Attend, Infer, Repeat: Fast Scene Understanding with Generative Modelscited by 209
- 2020SPACE: Unsupervised Object-Oriented Scene Representation via Spatial Attention and Decompositioncited by 190
- 2017Neural Expectation Maximizationcited by 123
- 2016Hierarchical Multiscale Recurrent Neural Networkscited by 102
- 2018Relational Neural Expectation Maximization: Unsupervised Discovery of Objects and their Interactionscited by 100
- 2016Tagger: Deep Unsupervised Perceptual Groupingcited by 69
- 2020Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challengecited by 21
- 2016A Joint Model for Word Embedding and Word Morphologycited by 14
- 2018Unsupervised Neural Word Segmentation for Chinese via Segmental Language Modelingcited by 12
- 2015An Unsupervised Method for Uncovering Morphological Chainscited by 8
- 2017Learning to Create and Reuse Words in Open-Vocabulary Neural Language Modelingcited by 8
- 2019A Latent Morphology Model for Open-Vocabulary Neural Machine Translationcited by 8
- 2020On Sparsifying Encoder Outputs in Sequence-to-Sequence Modelscited by 8
- 2017Unsupervised Learning of Morphological Forestscited by 2
Built on by
Citing papers per year: 2023: 2.
2 papers.