Kathleen: Oscillator-Based Byte-Level Text Classification Without Tokenization or Attention
Not on the map. Why?
Abstract (arXiv)
We present Kathleen, a text classification architecture that operates directly on raw UTF-8 bytes using frequency-domain processing -- requiring no tokenizer, no attention mechanism, and under 470K parameters. Kathleen introduces several novel components: (1) RecurrentOscillatorBanks -- damped sinusoid convolutions with temporal memory for O(L) sequence processing; (2) an FFT-Rotate Wavetable Encoder that maps all 256 byte values using a single learnable vector (256 floats); (3) PhaseHarmonics -- a sinusoidal non-linearity with just 6 learnable phase parameters (+2.6% accuracy, <0.001% of model parameters); (4) Content-Dependent Reverb with Positional Decay Modulation -- a temporal memory mechanism whose decay rate is jointly conditioned on input content and a learned position-indexed bias vector; (5) Token-Level Module Sequencer with consonance and dissonance interference channels. Through iterative architecture evolution from an initial 733K-parameter baseline (Kathleen-Clean) to the current Kathleen-V9 (469K parameters), we demonstrate that pretraining can be entirely eliminated while improving accuracy. Kathleen-V9 achieves 88.5% +/- 0.2% on IMDB, 92.4% +/- 0.2% on AG News, and 85.8% +/- 0.5% on SST-2 (3-seed averages) -- matching or exceeding the pretrained baseline on all benchmarks with 36% fewer parameters. On SST-2, the improvement is +2.5% absolute over the pretrained predecessor. Kathleen processes sequences in O(L) time and memory.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 21 references that are in the map. First posted on arXiv 9 Apr 2026.
Builds on
15 of 21 references.
References · 15
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,616
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2020Fourier Neural Operator for Parametric Partial Differential Equationscited by 2,806
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
- 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
- 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
- 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
- 2016Bag of Tricks for Efficient Text Classificationcited by 869
- 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 504
Built on by
Citing papers per year: 2026: 1.
1 paper.