# Transformer-Squared: Self-adaptive LLMs

arXiv: https://arxiv.org/abs/2501.06252 · HTML: https://mlsanity.com/paper/2501.06252/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Jan 2025.

## Built on by (most cited first, 16 of 16)

- [Self-Adapting Language Models](https://mlsanity.com/paper/2506.10943/) (2025)
- [EigenLoRAx: Recycling Adapters to Find Principal Subspaces for Resource-Efficient Adaptation and Inference](https://mlsanity.com/paper/2502.04700/) (2025)
- [Memory-Augmented Transformers: A Systematic Review from Neuroscience Principles to Enhanced Model Architectures](https://mlsanity.com/paper/2508.10824/) (2025)
- [ESSA: Evolutionary Strategies for Scalable Alignment](https://mlsanity.com/paper/2507.04453/) (2025)
- [MASS: MoErging through Adaptive Subspace Selection](https://mlsanity.com/paper/2504.05342/) (2025)
- [A quantum semantic framework for natural language processing](https://mlsanity.com/paper/2506.10077/) (2025)
- [TRINITY: An Evolved LLM Coordinator](https://mlsanity.com/paper/2512.04695/) (2025)
- [Sakana Fugu Technical Report](https://mlsanity.com/paper/2606.21228/) (2026)
- [Learning Composable Chains-of-Thought](https://mlsanity.com/paper/2505.22635/) (2025)
- [Do All Individual Layers Help? An Empirical Study of Task-Interfering Layers in Vision-Language Models](https://mlsanity.com/paper/2602.01167/) (2026)
- [Self-Consolidating Language Models: Continual Knowledge Incorporation from Context](https://mlsanity.com/paper/2605.07076/) (2026)
- [Eigenvectors of Experts are Training-free Non-collapsing Routers](https://mlsanity.com/paper/2605.30992/) (2026)
- [Efficient Multi-Source Knowledge Transfer by Model Merging](https://mlsanity.com/paper/2508.19353/) (2025)
- [PEML: Parameter-efficient Multi-Task Learning with Optimized Continuous Prompts](https://mlsanity.com/paper/2605.14055/) (2026)
- [PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Models](https://mlsanity.com/paper/2605.24549/) (2026)
- [Compositional Generalization in Autoregressive Models via Logit Composition](https://mlsanity.com/paper/2605.28304/) (2026)

## Abstract (arXiv)

Self-adaptive large language models (LLMs) aim to solve the challenges posed by traditional fine-tuning methods, which are often computationally intensive and static in their ability to handle diverse tasks. We introduce Transformer-Squared, a novel self-adaptation framework that adapts LLMs for unseen tasks in real-time by selectively adjusting only the singular components of their weight matrices. During inference, Transformer-Squared employs a two-pass mechanism: first, a dispatch system identifies the task properties, and then task-specific 'expert' vectors, trained using reinforcement learning, are dynamically mixed to obtain targeted behavior for the incoming prompt. Our method consistently outperforms ubiquitous approaches such as LoRA, with fewer parameters and greater efficiency. Furthermore, Transformer-Squared demonstrates versatility across different LLM architectures and modalities, including vision-language tasks. Transformer-Squared represents a significant leap forward, offering a scalable, efficient solution for enhancing the adaptability and task-specific performance of LLMs, paving the way for truly dynamic, self-organizing AI systems.