Signal in Noise: Exploring Meaning Encoded in Random Character Sequences with Character-Aware Language Models
Not on the map. Why?
Abstract (arXiv)
Natural language processing models learn word representations based on the distributional hypothesis, which asserts that word context (e.g., co-occurrence) correlates with meaning. We propose that n-grams composed of random character sequences, or garble, provide a novel context for studying word meaning both within and beyond extant language. In particular, randomly generated character n-grams lack meaning but contain primitive information based on the distribution of characters they contain. By studying the embeddings of a large corpus of garble, extant language, and pseudowords using CharacterBERT, we identify an axis in the model's high-dimensional embedding space that separates these classes of n-grams. Furthermore, we show that this axis relates to structure within extant language, including word part-of-speech, morphology, and concept concreteness. Thus, in contrast to studies that are mainly limited to extant language, our work reveals that meaning and primitive information are intrinsically linked.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 20 of its 59 references that are in the map. First posted on arXiv 15 Mar 2022.
Builds on
20 of 59 references.
References · 20
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
- 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
- 2018Deep contextualized word representationscited by 2,285
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2020A Primer in BERTology: What we know about how BERT workscited by 807
- 2019How Contextual are Contextualized Word Representations? Comparing the Geometry of BERT, ELMo, and GPT-2 Embeddingscited by 770
- 2019Assessing BERT's Syntactic Abilitiescited by 191
- 2019Text Classification Algorithms: A Surveycited by 163
- 2018Character-Level Language Modeling with Deeper Self-Attentioncited by 138
- 2015Character-Aware Neural Language Modelscited by 129
8 more
- 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
- 2020CharBERT: Character-aware Pre-trained Language Modelcited by 63
- 2021Superbizarre Is Not Superb: Derivational Morphology Improves BERT's Interpretation of Complex Wordscited by 63
- 2018A Multilayer Convolutional Encoder-Decoder Neural Network for Grammatical Error Correctioncited by 36
- 2016Neural Language Correction with Character-Based Attentioncited by 18
- 2020What are the Goals of Distributional Semantics?cited by 8
- 2020comp-syn: Perceptually Grounded Word Embeddings with Colorcited by 3
- 2020The curious case of developmental BERTology: On sparsity, transfer learning, generalization and the braincited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.