# Syntax Representation in Word Embeddings and Neural Networks -- A Survey

arXiv: https://arxiv.org/abs/2010.01063 · HTML: https://mlsanity.com/paper/2010.01063/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Oct 2020.

## Built on by (most cited first, 5 of 5)

- [Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting](https://mlsanity.com/paper/2406.00053/) (2024)
- [Literature review on vulnerability detection using NLP technology](https://mlsanity.com/paper/2104.11230/) (2021)
- [Emergent Linguistic Structures in Neural Networks are Fragile](https://mlsanity.com/paper/2210.17406/) (2022)
- [Can We Use Probing to Better Understand Fine-tuning and Knowledge Distillation of the BERT NLU?](https://mlsanity.com/paper/2301.11688/) (2023)
- [Urdu Dependency Parsing and Treebank Development: A Syntactic and Morphological Perspective](https://mlsanity.com/paper/2406.09549/) (2024)

## Abstract (arXiv)

Neural networks trained on natural language processing tasks capture syntax even though it is not provided as a supervision signal. This indicates that syntactic analysis is essential to the understating of language in artificial intelligence systems. This overview paper covers approaches of evaluating the amount of syntactic information included in the representations of words for different neural network architectures. We mainly summarize re-search on English monolingual data on language modeling tasks and multilingual data for neural machine translation systems and multilingual language models. We describe which pre-trained models and representations of language are best suited for transfer to syntactic tasks.