# Towards Signal Processing In Large Language Models

arXiv: https://arxiv.org/abs/2406.10254 · HTML: https://mlsanity.com/paper/2406.10254/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 41 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [CNN Architectures for Large-Scale Audio Classification](https://mlsanity.com/paper/1609.09430/) (2016)
- [Tacotron: Towards End-to-End Speech Synthesis](https://mlsanity.com/paper/1703.10135/) (2017)
- [Conv-TasNet: Surpassing Ideal Time-Frequency Magnitude Masking for Speech Separation](https://mlsanity.com/paper/1809.07454/) (2018)
- [Pix2seq: A Language Modeling Framework for Object Detection](https://mlsanity.com/paper/2109.10852/) (2021)
- [FNet: Mixing Tokens with Fourier Transforms](https://mlsanity.com/paper/2105.03824/) (2021)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Speaker Recognition from Raw Waveform with SincNet](https://mlsanity.com/paper/1808.00158/) (2018)
- [Character-Level Language Modeling with Deeper Self-Attention](https://mlsanity.com/paper/1808.04444/) (2018)
- [ProGen: Language Modeling for Protein Generation](https://mlsanity.com/paper/2004.03497/) (2020)
- [PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregation](https://mlsanity.com/paper/2102.01243/) (2021)
- [Audio Transformers](https://mlsanity.com/paper/2105.00335/) (2021)
- [Why ResNet Works? Residuals Generalize](https://mlsanity.com/paper/1904.01367/) (2019)
- [Language Through a Prism: A Spectral Approach for Multiscale Language Representations](https://mlsanity.com/paper/2011.04823/) (2020)
- [A Framework for Generative and Contrastive Learning of Audio Representations](https://mlsanity.com/paper/2010.11459/) (2020)
- [Learning Multiscale Features Directly From Waveforms](https://mlsanity.com/paper/1603.09509/) (2016)

## Built on by (most cited first, 7 of 7)

- [Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2504.14891/) (2025)
- [DeepSeek-Inspired Exploration of RL-based LLMs and Synergy with Wireless Networks: A Survey](https://mlsanity.com/paper/2503.09956/) (2025)
- [Energy-Gated Attention: Spectral Salience as an Inductive Bias for Transformer Attention](https://mlsanity.com/paper/2605.21842/) (2026)
- [Multimodal-to-Text Prompt Engineering in Large Language Models Using Feature Embeddings for GNSS Interference Characterization](https://mlsanity.com/paper/2501.05079/) (2025)
- [Energy-Gated Attention and Wavelet Positional Encoding: Complementary Inductive Biases for Transformer Attention](https://mlsanity.com/paper/2605.26355/) (2026)
- [Beyond Sinusoids: A Morlet Wavelet Framework for Transformer Positional Encoding](https://mlsanity.com/paper/2606.01258/) (2026)
- [Multiscale POD of Transformer Attention Fields: Scale-Selective Analysis via Morlet Scalogram](https://mlsanity.com/paper/2606.06573/) (2026)

## Abstract (arXiv)

This paper introduces the idea of applying signal processing inside a Large Language Model (LLM). With the recent explosion of generative AI, our work can help bridge two fields together, namely the field of signal processing and large language models. We draw parallels between classical Fourier-Transforms and Fourier Transform-like learnable time-frequency representations for every intermediate activation signal of an LLM. Once we decompose every activation signal across tokens into a time-frequency representation, we learn how to filter and reconstruct them, with all components learned from scratch, to predict the next token given the previous context. We show that for GPT-like architectures, our work achieves faster convergence and significantly increases performance by adding a minuscule number of extra parameters when trained for the same epochs. We hope this work paves the way for algorithms exploring signal processing inside the signals found in neural architectures like LLMs and beyond.