# The Credibility Transformer

arXiv: https://arxiv.org/abs/2409.16653 · HTML: https://mlsanity.com/paper/2409.16653/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 25 Sep 2024.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [GLU Variants Improve Transformer](https://mlsanity.com/paper/2002.05202/) (2020)
- [Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning](https://mlsanity.com/paper/1702.03118/) (2017)
- [Revisiting Deep Learning Models for Tabular Data](https://mlsanity.com/paper/2106.11959/) (2021)
- [Language Modeling with Gated Convolutional Networks](https://mlsanity.com/paper/1612.08083/) (2016)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [On Embeddings for Numerical Features in Tabular Deep Learning](https://mlsanity.com/paper/2203.05556/) (2022)
- [Better by Default: Strong Pre-Tuned MLPs and Boosted Trees on Tabular Data](https://mlsanity.com/paper/2407.04491/) (2024)
- [Entity Embeddings of Categorical Variables](https://mlsanity.com/paper/1604.06737/) (2016)
- [NormFormer: Improved Transformer Pretraining with Extra Normalization](https://mlsanity.com/paper/2110.09456/) (2021)
- [Artificial Neural Networks Applied to Taxi Destination Prediction](https://mlsanity.com/paper/1508.00021/) (2015)
- [LocalGLMnet: interpretable deep learning for tabular data](https://mlsanity.com/paper/2107.11059/) (2021)
- [Enhancing Actuarial Non-Life Pricing Models via Transformers](https://mlsanity.com/paper/2311.07597/) (2023)

## Built on by (most cited first, 5 of 5)

- [Tab-TRM: Tiny Recursive Model for Insurance Pricing on Tabular Data](https://mlsanity.com/paper/2601.07675/) (2026)
- [Tree-like Pairwise Interaction Networks](https://mlsanity.com/paper/2508.15678/) (2025)
- [From Chain-Ladder to Individual Claims Reserving](https://mlsanity.com/paper/2602.15385/) (2026)
- [Is TabPFN the Silver Bullet for Insurance Pricing?](https://mlsanity.com/paper/2605.22892/) (2026)
- [Many Optimizers But Only One Training Path: Repeated Resampling for Adaptive Optimizer Selection](https://mlsanity.com/paper/2608.18810/) (2026)

## Abstract (arXiv)

Inspired by the large success of Transformers in Large Language Models, these architectures are increasingly applied to tabular data. This is achieved by embedding tabular data into low-dimensional Euclidean spaces resulting in similar structures as time-series data. We introduce a novel credibility mechanism to this Transformer architecture. This credibility mechanism is based on a special token that should be seen as an encoder that consists of a credibility weighted average of prior information and observation based information. We demonstrate that this novel credibility mechanism is very beneficial to stabilize training, and our Credibility Transformer leads to predictive models that are superior to state-of-the-art deep learning models.