# Language Through a Prism: A Spectral Approach for Multiscale Language Representations

arXiv: https://arxiv.org/abs/2011.04823 · HTML: https://mlsanity.com/paper/2011.04823/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2020.

## Built on by (most cited first, 30 of 39)

- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [Continuous diffusion for categorical data](https://mlsanity.com/paper/2211.15089/) (2022)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [Audio Transformers](https://mlsanity.com/paper/2105.00335/) (2021)
- [Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization](https://mlsanity.com/paper/2412.17739/) (2024)
- [DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space](https://mlsanity.com/paper/2412.15032/) (2024)
- [Task Ambiguity in Humans and Language Models](https://mlsanity.com/paper/2212.10711/) (2022)
- [Language modeling via stochastic processes](https://mlsanity.com/paper/2203.11370/) (2022)
- [Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers](https://mlsanity.com/paper/2302.01925/) (2023)
- [Deep Autoregressive Models with Spectral Attention](https://mlsanity.com/paper/2107.05984/) (2021)
- [Feature Dropout: Revisiting the Role of Augmentations in Contrastive Learning](https://mlsanity.com/paper/2212.08378/) (2022)
- [Vim-F: Visual State Space Model Benefiting from Learning in the Frequency Domain](https://mlsanity.com/paper/2405.18679/) (2024)
- [Towards Signal Processing In Large Language Models](https://mlsanity.com/paper/2406.10254/) (2024)
- [Extracting Sentence Embeddings from Pretrained Transformer Models](https://mlsanity.com/paper/2408.08073/) (2024)
- [A quantitative analysis of semantic information in deep representations of text and images](https://mlsanity.com/paper/2505.17101/) (2025)
- [Differential syntactic and semantic encoding in LLMs](https://mlsanity.com/paper/2601.04765/) (2026)
- [Fast-FNet: Accelerating Transformer Encoder Models via Efficient Fourier Layers](https://mlsanity.com/paper/2209.12816/) (2022)
- [MGT-Prism: Enhancing Domain Generalization for Machine-Generated Text Detection via Spectral Alignment](https://mlsanity.com/paper/2508.13768/) (2025)
- [Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention](https://mlsanity.com/paper/2605.21842/) (2026)
- [Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention](https://mlsanity.com/paper/2605.26355/) (2026)
- [SpectralCLIP: Preventing Artifacts in Text-Guided Style Transfer from a Spectral Perspective](https://mlsanity.com/paper/2303.09270/) (2023)
- [Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding](https://mlsanity.com/paper/2606.01258/) (2026)
- [Oscillatory Fourier Neural Network: A Compact and Efficient Architecture for Sequential Processing](https://mlsanity.com/paper/2109.13090/) (2021)
- [Exploiting Inter-Session Information with Frequency-enhanced Dual-Path Networks for Sequential Recommendation](https://mlsanity.com/paper/2511.06285/) (2025)
- [MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling](https://mlsanity.com/paper/2512.23824/) (2025)
- [Spectral Probing](https://mlsanity.com/paper/2210.11860/) (2022)
- [Conformer LLMs -- Convolution Augmented Large Language Models](https://mlsanity.com/paper/2307.00461/) (2023)
- [Repurformer: Transformers for Repurposing-Aware Molecule Generation](https://mlsanity.com/paper/2407.11439/) (2024)
- [Wavelet GPT: Wavelet Inspired Large Language Models](https://mlsanity.com/paper/2409.12924/) (2024)
- [A Free Probabilistic Framework for Analyzing the Transformer-based Language Models](https://mlsanity.com/paper/2506.16550/) (2025)

## Abstract (arXiv)

Language exhibits structure at different scales, ranging from subwords to words, sentences, paragraphs, and documents. To what extent do deep models capture information at these scales, and can we force them to better capture structure across this hierarchy? We approach this question by focusing on individual neurons, analyzing the behavior of their activations at different timescales. We show that signal processing provides a natural framework for separating structure across scales, enabling us to 1) disentangle scale-specific information in existing embeddings and 2) train models to learn more about particular scales. Concretely, we apply spectral filters to the activations of a neuron across an input, producing filtered embeddings that perform well on part of speech tagging (word-level), dialog speech acts classification (utterance-level), or topic classification (document-level), while performing poorly on the other tasks. We also present a prism layer for training models, which uses spectral filters to constrain different neurons to model structure at different scales. Our proposed BERT + Prism model can better predict masked tokens using long-range context and produces multiscale representations that perform better at utterance- and document-level tasks. Our methods are general and readily applicable to other domains besides language, such as images, audio, and video.