# From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction

arXiv: https://arxiv.org/abs/2511.12081 · HTML: https://mlsanity.com/paper/2511.12081/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Placed in

- LLM reasoning / [Test-time compute and efficient thinking](https://mlsanity.com/lr/test-time-compute-and-efficient-thinking/)

## Builds on (most cited first, 21 of 21)

- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Attentive Sequential Recommendation](https://mlsanity.com/paper/1808.09781/) (2018)
- [BERT4Rec: Sequential Recommendation with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1904.06690/) (2019)
- [DeepFM: A Factorization-Machine based Neural Network for CTR Prediction](https://mlsanity.com/paper/1703.04247/) (2017)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems](https://mlsanity.com/paper/2008.13535/) (2020)
- [AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://mlsanity.com/paper/1810.11921/) (2018)
- [Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations](https://mlsanity.com/paper/2402.17152/) (2024)
- [Wukong: Towards a Scaling Law for Large-Scale Recommendation](https://mlsanity.com/paper/2403.02545/) (2024)
- [HLLM: Enhancing Sequential Recommendations via Hierarchical Large Language Models for Item and User Modeling](https://mlsanity.com/paper/2409.12740/) (2024)
- [Field-weighted Factorization Machines for Click-Through Rate Prediction in Display Advertising](https://mlsanity.com/paper/1806.03514/) (2018)
- [RankMixer: Scaling Up Ranking Models in Industrial Recommenders](https://mlsanity.com/paper/2507.15551/) (2025)
- [Analysing The Impact of Sequence Composition on Language Model Pre-Training](https://mlsanity.com/paper/2402.13991/) (2024)
- [Multi-Behavior Generative Recommendation](https://mlsanity.com/paper/2405.16871/) (2024)
- [Inductive Generative Recommendation via Retrieval-based Speculation](https://mlsanity.com/paper/2410.02939/) (2024)
- [Unlocking Scaling Law in Industrial Recommendation Systems with a Three-step Paradigm based Large User Model](https://mlsanity.com/paper/2502.08309/) (2025)
- [Hiformer: Heterogeneous Feature Interactions Learning with Transformers for Recommender Systems](https://mlsanity.com/paper/2311.05884/) (2023)
- [Scaling New Frontiers: Insights into Large Recommendation Models](https://mlsanity.com/paper/2412.00714/) (2024)
- [Scaling Transformers for Discriminative Recommendation via Generative Pretraining](https://mlsanity.com/paper/2506.03699/) (2025)
- [Explicit Semantic Cross Feature Learning via Pre-trained Graph Neural Networks for CTR Prediction](https://mlsanity.com/paper/2105.07752/) (2021)

## Built on by (most cited first, 3 of 3)

- [UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems](https://mlsanity.com/paper/2604.00590/) (2026)
- [MDL: A Unified Multi-Distribution Learner in Large-scale Industrial Recommendation through Tokenization](https://mlsanity.com/paper/2602.07520/) (2026)
- [SpecFormer: Mitigating Embedding and Attention Collapse via Spectral-Aware Transformer for Recommendation](https://mlsanity.com/paper/2607.24025/) (2026)

## Abstract (arXiv)

Despite massive investments in scale, deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns -- a stark contrast to the predictable scaling laws seen in large language models (LLMs). We identify the root cause as a fundamental textitstructural misalignment: standard Transformers assume sequential compositionality, whereas CTR data demand combinatorial reasoning over heterogeneous fields. To restore alignment, we introduce the Field-Aware Transformer (FAT). By reconstructing the standard Transformer block with field-centric parameters, FAT achieves textitstructured expressivity, fundamentally shifting the model complexity dependence from the total vocabulary size n with the number of fields F (n gg F). Crucially, to decouple model capacity from field cardinality, FAT employs a Basis-Composed Hypernetwork to synthesize field-specific parameters from shared bases, further reducing parameter complexity. Theoretically, we ground this scaling behavior through a formal scaling law based on Rademacher complexity. Empirically, FAT outperforms exisiting state-of-the-art methods with up to textbf+4.38% AUC improvement, and delivers +2.33% CTR and +0.66% RPM in live production. Our work establishes that scalable recommendation arises not from size alone, but from textitstructured expressivity -- architectural coherence with data semantics.