# Correction of Transformer-Based Models with Smoothing Pseudo-Projector

arXiv: https://arxiv.org/abs/2603.09815 · HTML: https://mlsanity.com/paper/2603.09815/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 10 Mar 2026.

## Builds on (most cited first, 6 of 6)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Learning Algebraic Multigrid Using Graph Neural Networks](https://mlsanity.com/paper/2003.05744/) (2020)
- [MgNet: A Unified Framework of Multigrid and Convolutional Neural Network](https://mlsanity.com/paper/1901.10415/) (2019)
- [Layer-Parallel Training of Deep Residual Neural Networks](https://mlsanity.com/paper/1812.04352/) (2018)
- [M2NO: An Efficient Multi-Resolution Operator Framework for Dynamic Multi-Scale PDE Solvers](https://mlsanity.com/paper/2406.04822/) (2024)
- [Parallel Training of GRU Networks with a Multi-Grid Solver for Long Sequences](https://mlsanity.com/paper/2203.04738/) (2022)

## Abstract (arXiv)

The pseudo-projector is a lightweight modification that can be integrated into existing language models and other neural networks without altering their core architecture. It can be viewed as a hidden-representation corrector that reduces sensitivity to noise by suppressing directions induced by label-irrelevant input content. The design is inspired by the multigrid (MG) paradigm, originally developed to accelerate the convergence of iterative solvers for partial differential equations and boundary value problems, and later extended to more general linear systems through algebraic multigrid methods. We refer to the method as a pseudo-projector because its linear prototype corresponds to a strictly idempotent orthogonal projector, whereas the practical formulation employs learnable restriction and prolongation operators and therefore does not, in general, satisfy the properties of an exact orthogonal projection. We evaluate the proposed approach on transformer-based text classification tasks, as well as controlled synthetic benchmarks, demonstrating its effectiveness in improving training dynamics and robustness. Experimental results, together with supporting theoretical heuristics, indicate consistent improvements in training behavior across a range of settings, with no adverse effects observed otherwise. Our next step will be to extend this approach to language models.