# Can a Fruit Fly Learn Word Embeddings?

arXiv: https://arxiv.org/abs/2101.06887 · HTML: https://mlsanity.com/paper/2101.06887/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 18 Jan 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations](https://mlsanity.com/paper/1808.09121/) (2018)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [HDLTex: Hierarchical Deep Learning for Text Classification](https://mlsanity.com/paper/1709.08267/) (2017)
- [Unsupervised Learning by Competing Hidden Units](https://mlsanity.com/paper/1806.10181/) (2018)
- [Local Unsupervised Learning for Image Analysis](https://mlsanity.com/paper/1908.08993/) (2019)
- [Near-lossless Binarization of Word Embeddings](https://mlsanity.com/paper/1803.09065/) (2018)
- [Card-660: Cambridge Rare Word Dataset - a Reliable Benchmark for Infrequent Word Representation Models](https://mlsanity.com/paper/1808.09308/) (2018)
- [Bio-Inspired Hashing for Unsupervised Similarity Search](https://mlsanity.com/paper/2001.04907/) (2020)

## Built on by (most cited first, 17 of 17)

- [FlyLoRA: Boosting Task Decoupling and Parameter Efficiency via Implicit Rank-Wise Mixture-of-Experts](https://mlsanity.com/paper/2510.08396/) (2025)
- [Memory in Plain Sight: Surveying the Uncanny Resemblances of Associative Memories and Diffusion Models](https://mlsanity.com/paper/2309.16750/) (2023)
- [Sparse Distributed Memory is a Continual Learner](https://mlsanity.com/paper/2303.11934/) (2023)
- [Fly-CL: A Fly-Inspired Framework for Enhancing Efficient Decorrelation and Reduced Training Time in Pre-trained Model-based Continual Representation Learning](https://mlsanity.com/paper/2510.16877/) (2025)
- [DINE: Dimensional Interpretability of Node Embeddings](https://mlsanity.com/paper/2310.01162/) (2023)
- [Federated Nearest Neighbor Classification with a Colony of Fruit-Flies: With Supplement](https://mlsanity.com/paper/2112.07157/) (2021)
- [Learning to Modulate Random Weights: Neuromodulation-inspired Neural Networks For Efficient Continual Learning](https://mlsanity.com/paper/2204.04297/) (2022)
- [Using Connectome Features to Constrain Echo State Networks](https://mlsanity.com/paper/2206.02094/) (2022)
- [Hebbian Continual Representation Learning](https://mlsanity.com/paper/2207.04874/) (2022)
- [Language Cognition and Language Computation -- Human and Machine Language Understanding](https://mlsanity.com/paper/2301.04788/) (2023)
- [SHARP: Sparsity and Hidden Activation RePlay for Neuro-Inspired Continual Learning](https://mlsanity.com/paper/2305.18563/) (2023)
- [On Design Choices in Similarity-Preserving Sparse Randomized Embeddings](https://mlsanity.com/paper/2501.14741/) (2024)
- [Active Inference in Hebbian Learning Networks](https://mlsanity.com/paper/2306.05053/) (2023)
- [SNNLP: Energy-Efficient Natural Language Processing Using Spiking Neural Networks](https://mlsanity.com/paper/2401.17911/) (2024)
- [Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfaction](https://mlsanity.com/paper/2502.01706/) (2025)
- [Biological Processing Units: Leveraging an Insect Connectome to Pioneer Biofidelic Neural Architectures](https://mlsanity.com/paper/2507.10951/) (2025)
- [From Classification to Regression: Using a Fruitfly to Solve Equations](https://mlsanity.com/paper/2607.27196/) (2026)

## Abstract (arXiv)

The mushroom body of the fruit fly brain is one of the best studied systems in neuroscience. At its core it consists of a population of Kenyon cells, which receive inputs from multiple sensory modalities. These cells are inhibited by the anterior paired lateral neuron, thus creating a sparse high dimensional representation of the inputs. In this work we study a mathematical formalization of this network motif and apply it to learning the correlational structure between words and their context in a corpus of unstructured text, a common natural language processing (NLP) task. We show that this network can learn semantic representations of words and can generate both static and context-dependent word embeddings. Unlike conventional methods (e.g., BERT, GloVe) that use dense representations for word embedding, our algorithm encodes semantic meaning of words and their context in the form of sparse binary hash codes. The quality of the learned representations is evaluated on word similarity analysis, word-sense disambiguation, and document classification. It is shown that not only can the fruit fly network motif achieve performance comparable to existing methods in NLP, but, additionally, it uses only a fraction of the computational resources (shorter training time and smaller memory footprint).