# Language Modeling Teaches You More Syntax than Translation Does: Lessons Learned Through Auxiliary Task Analysis

arXiv: https://arxiv.org/abs/1809.10040 · HTML: https://mlsanity.com/paper/1809.10040/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 Sep 2018.

## Built on by (most cited first, 17 of 17)

- [SAEs Are Good for Steering -- If You Select the Right Features](https://mlsanity.com/paper/2505.20063/) (2025)
- [Conditional probing: measuring usable information beyond a baseline](https://mlsanity.com/paper/2109.09234/) (2021)
- [Beyond Surface Statistics: Scene Representations in a Latent Diffusion Model](https://mlsanity.com/paper/2306.05720/) (2023)
- [Knowing Before Saying: LLM Representations Encode Information About Chain-of-Thought Success Before Completion](https://mlsanity.com/paper/2505.24362/) (2025)
- [Grounding Natural Language Instructions: Can Large Language Models Capture Spatial Information?](https://mlsanity.com/paper/2109.08634/) (2021)
- [Does My Representation Capture X? Probe-Ably](https://mlsanity.com/paper/2104.05807/) (2021)
- [Morphosyntactic probing of multilingual BERT models](https://mlsanity.com/paper/2306.06205/) (2023)
- [Probing Linguistic Information For Logical Inference In Pre-trained Language Models](https://mlsanity.com/paper/2112.01753/) (2021)
- [Probing with Noise: Unpicking the Warp and Weft of Embeddings](https://mlsanity.com/paper/2210.12206/) (2022)
- [Idioms, Probing and Dangerous Things: Towards Structural Probing for Idiomaticity in Vector Space](https://mlsanity.com/paper/2304.14333/) (2023)
- [Low Resource Summarization using Pre-trained Language Models](https://mlsanity.com/paper/2310.02790/) (2023)
- [EigenNoise: A Contrastive Prior to Warm-Start Representations](https://mlsanity.com/paper/2205.04376/) (2022)
- [Enhancing E-Commerce Recommendation using Pre-Trained Language Model and Fine-Tuning](https://mlsanity.com/paper/2302.04443/) (2023)
- [Foundation Model's Embedded Representations May Detect Distribution Shift](https://mlsanity.com/paper/2310.13836/) (2023)
- [Emergent Manifold Separability during Reasoning in Large Language Models](https://mlsanity.com/paper/2602.20338/) (2026)
- [Knowledge-Aware Neural Networks for Medical Forum Question Classification](https://mlsanity.com/paper/2109.13141/) (2021)
- [StructFormer: Document Structure-based Masked Attention and its Impact on Language Model Pre-Training](https://mlsanity.com/paper/2411.16618/) (2024)

## Abstract (arXiv)

Recent work using auxiliary prediction task classifiers to investigate the properties of LSTM representations has begun to shed light on why pretrained representations, like ELMo (Peters et al., 2018) and CoVe (McCann et al., 2017), are so beneficial for neural language understanding models. We still, though, do not yet have a clear understanding of how the choice of pretraining objective affects the type of linguistic information that models learn. With this in mind, we compare four objectives---language modeling, translation, skip-thought, and autoencoding---on their ability to induce syntactic and part-of-speech information. We make a fair comparison between the tasks by holding constant the quantity and genre of the training data, as well as the LSTM architecture. We find that representations from language models consistently perform best on our syntactic auxiliary prediction tasks, even when trained on relatively small amounts of data. These results suggest that language modeling may be the best data-rich pretraining task for transfer learning applications requiring syntactic information. We also find that the representations from randomly-initialized, frozen LSTMs perform strikingly well on our syntactic auxiliary tasks, but this effect disappears when the amount of training data for the auxiliary tasks is reduced.