# Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing

arXiv: https://arxiv.org/abs/2410.11462 · HTML: https://mlsanity.com/paper/2410.11462/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 57 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 8 of 8)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Whitening Sentence Representations for Better Semantics and Faster Retrieval](https://mlsanity.com/paper/2103.15316/) (2021)
- [Anisotropy Is Inherent to Self-Attention in Transformers](https://mlsanity.com/paper/2401.12143/) (2024)
- [Attentive Mimicking: Better Word Embeddings by Attending to Informative Contexts](https://mlsanity.com/paper/1904.01617/) (2019)
- [LERT: A Linguistically-motivated Pre-trained Language Model](https://mlsanity.com/paper/2211.05344/) (2022)

## Built on by (most cited first, 10 of 10)

- [Empirical Analysis of Decoding Biases in Masked Diffusion Models](https://mlsanity.com/paper/2508.13021/) (2025)
- [Once Upon a Time: Interactive Learning for Storytelling with Small Language Models](https://mlsanity.com/paper/2509.15714/) (2025)
- [Developmentally-plausible Working Memory Shapes a Critical Period for Language Acquisition](https://mlsanity.com/paper/2502.04795/) (2025)
- [Enhancing Text-to-Image Diffusion Transformer via Split-Text Conditioning](https://mlsanity.com/paper/2505.19261/) (2025)
- [Disentangling Geometry, Performance, and Training in Language Models](https://mlsanity.com/paper/2602.20433/) (2026)
- [Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics](https://mlsanity.com/paper/2604.08764/) (2026)
- [Relative Overfitting and Accept-Reject Framework](https://mlsanity.com/paper/2505.07783/) (2025)
- [Does Episodic Memory Help Close the Lexical Frequency Gap in Sensitivity to Syntactic Contrasts? A Test Using Retrieval-Augmented Language Models](https://mlsanity.com/paper/2608.23851/) (2026)
- [Visual Information-Guided Parallel Decoding for Diffusion Multimodal Large Language Models](https://mlsanity.com/paper/2608.26580/) (2026)
- [Understanding and Exploiting Anisotropy in Post-Training](https://mlsanity.com/paper/2609.32792/) (2026)

## Abstract (arXiv)

Language models strongly rely on frequency information because they maximize the likelihood of tokens during pre-training. As a consequence, language models tend to not generalize well to tokens that are seldom seen during training. Moreover, maximum likelihood training has been discovered to give rise to anisotropy: representations of tokens in a model tend to cluster tightly in a high-dimensional cone, rather than spreading out over their representational capacity. Our work introduces a method for quantifying the frequency bias of a language model by assessing sentence-level perplexity with respect to token-level frequency. We then present a method for reducing the frequency bias of a language model by inducing a syntactic prior over token representations during pre-training. Our Syntactic Smoothing method adjusts the maximum likelihood objective function to distribute the learning signal to syntactically similar tokens. This approach results in better performance on infrequent English tokens and a decrease in anisotropy. We empirically show that the degree of anisotropy in a model correlates with its frequency bias.