# Introduction to Transformers: an NLP Perspective

arXiv: https://arxiv.org/abs/2311.17633 · HTML: https://mlsanity.com/paper/2311.17633/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Nov 2023.

## Built on by (most cited first, 10 of 10)

- [A mathematical perspective on Transformers](https://mlsanity.com/paper/2312.10794/) (2023)
- [Efficient Prompting Methods for Large Language Models: A Survey](https://mlsanity.com/paper/2404.01077/) (2024)
- [GRAM: A Generative Foundation Reward Model for Reward Generalization](https://mlsanity.com/paper/2506.14175/) (2025)
- [RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data](https://mlsanity.com/paper/2408.12109/) (2024)
- [Cross-layer Attention Sharing for Pre-trained Large Language Models](https://mlsanity.com/paper/2408.01890/) (2024)
- [Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers](https://mlsanity.com/paper/2411.09224/) (2024)
- [Prior Constraints-based Reward Model Training for Aligning Large Language Models](https://mlsanity.com/paper/2404.00978/) (2024)
- [Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models](https://mlsanity.com/paper/2511.12464/) (2025)
- [A Novel Vision Transformer for Camera-LiDAR Fusion based Traffic Object Segmentation](https://mlsanity.com/paper/2501.02858/) (2025)
- [Precise Verification of Transformers through ReLU-Catalyzed Abstraction Refinement](https://mlsanity.com/paper/2605.14294/) (2026)

## Abstract (arXiv)

Transformers have dominated empirical machine learning models of natural language processing. In this paper, we introduce basic concepts of Transformers and present key techniques that form the recent advances of these models. This includes a description of the standard Transformer architecture, a series of model refinements, and common applications. Given that Transformers and related deep learning techniques might be evolving in ways we have never seen, we cannot dive into all the model details or cover all the technical areas. Instead, we focus on just those concepts that are helpful for gaining a good understanding of Transformers and their variants. We also summarize the key ideas that impact this field, thereby yielding some insights into the strengths and limitations of these models.