# The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectives

arXiv: https://arxiv.org/abs/1909.01380 · HTML: https://mlsanity.com/paper/1909.01380/

## Key facts

- Cited by 139 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2019.

## Built on by (most cited first, 30 of 139)

- [Do Vision Transformers See Like Convolutional Neural Networks?](https://mlsanity.com/paper/2108.08810/) (2021)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Space](https://mlsanity.com/paper/2203.14680/) (2022)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Layer by Layer: Uncovering Hidden Representations in Language Models](https://mlsanity.com/paper/2502.02013/) (2025)
- [LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding](https://mlsanity.com/paper/2404.16710/) (2024)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Transformer Language Models without Positional Encodings Still Learn Positional Information](https://mlsanity.com/paper/2203.16634/) (2022)
- [What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?](https://mlsanity.com/paper/2204.05832/) (2022)
- [Analyzing Transformers in Embedding Space](https://mlsanity.com/paper/2209.02535/) (2022)
- [Large Language Models in Law: A Survey](https://mlsanity.com/paper/2312.03718/) (2023)
- [Should You Mask 15% in Masked Language Modeling?](https://mlsanity.com/paper/2202.08005/) (2022)
- [Learning Graph Structures with Transformer for Multivariate Time Series Anomaly Detection in IoT](https://mlsanity.com/paper/2104.03466/) (2021)
- [Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue](https://mlsanity.com/paper/2401.04700/) (2024)
- [Language Models Implement Simple Word2Vec-style Vector Arithmetic](https://mlsanity.com/paper/2305.16130/) (2023)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Jump to Conclusions: Short-Cutting Transformers With Linear Transformations](https://mlsanity.com/paper/2303.09435/) (2023)
- [Neurons in Large Language Models: Dead, N-gram, Positional](https://mlsanity.com/paper/2309.04827/) (2023)
- [Representation Topology Divergence: A Method for Comparing Neural Network Representations](https://mlsanity.com/paper/2201.00058/) (2021)
- [What Do Self-Supervised Speech Models Know About Words?](https://mlsanity.com/paper/2307.00162/) (2023)
- [Factual Confidence of LLMs: on Reliability and Robustness of Current Estimators](https://mlsanity.com/paper/2406.13415/) (2024)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [From Yes-Men to Truth-Tellers: Addressing Sycophancy in Large Language Models with Pinpoint Tuning](https://mlsanity.com/paper/2409.01658/) (2024)
- [Variational Information Bottleneck for Effective Low-Resource Fine-Tuning](https://mlsanity.com/paper/2106.05469/) (2021)
- [Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer](https://mlsanity.com/paper/2205.11631/) (2022)
- [Fantastic Semantics and Where to Find Them: Investigating Which Layers of Generative LLMs Reflect Lexical Semantics](https://mlsanity.com/paper/2403.01509/) (2024)
- [Privacy-Preserving Parameter-Efficient Fine-Tuning for Large Language Model Services](https://mlsanity.com/paper/2305.06212/) (2023)
- [Explaining How Transformers Use Context to Build Predictions](https://mlsanity.com/paper/2305.12535/) (2023)
- [The Low-Dimensional Linear Geometry of Contextualized Word Representations](https://mlsanity.com/paper/2105.07109/) (2021)

## Abstract (arXiv)

We seek to understand how the representations of individual tokens and the structure of the learned feature space evolve between layers in deep neural networks under different learning objectives. We focus on the Transformers for our analysis as they have been shown effective on various tasks, including machine translation (MT), standard left-to-right language models (LM) and masked language modeling (MLM). Previous work used black-box probing tasks to show that the representations learned by the Transformer differ significantly depending on the objective. In this work, we use canonical correlation analysis and mutual information estimators to study how information flows across Transformer layers and how this process depends on the choice of learning objective. For example, as you go from bottom to top layers, information about the past in left-to-right language models gets vanished and predictions about the future get formed. In contrast, for MLM, representations initially acquire information about the context around the token, partially forgetting the token identity and producing a more generalized token representation. The token identity then gets recreated at the top MLM layers.