mlsanityData to 7 Oct 2026

The Hidden Space of Transformer Language Adapters

arXiv 2402.13137 ↗First posted 20 Feb 2024cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

We analyze the operation of transformer language adapters, which are small modules trained on top of a frozen language model to adapt its predictions to new target languages. We show that adapted predictions mostly evolve in the source language the model was trained on, while the target language becomes pronounced only in the very last layers of the model. Moreover, the adaptation process is gradual and distributed across layers, where it is possible to skip small groups of adapters without decreasing adaptation performance. Last, we show that adapters operate on top of the model's frozen representation space while largely preserving its structure, rather than on an 'isolated' subspace. Our findings provide a deeper view into the adaptation process of language models to new languages, showcasing the constraints imposed on it by the underlying model and introduces practical implications to enhance its efficiency.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 44 of its 58 references that are in the map. First posted on arXiv 20 Feb 2024.

Builds on

44 of 58 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  10. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2019How multilingual is Multilingual BERT?cited by 642
32 more
  1. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  2. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
  3. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  6. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  7. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  8. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  9. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  10. 2023In-Context Learning Creates Task Vectorscited by 302
  11. 2020InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Trainingcited by 237
  12. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  13. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  14. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  15. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  16. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
  17. 2019Neural Machine Translation with Byte-Level Subwordscited by 115
  18. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  19. 2022The Geometry of Multilingual Language Model Representationscited by 94
  20. 2023Jump to Conclusions: Short-Cutting Transformers With Linear Transformationscited by 89
  21. 2020On the Language Neutrality of Pre-trained Multilingual Representationscited by 82
  22. 2021First Align, then Predict: Understanding the Cross-Lingual Ability of Multilingual BERTcited by 64
  23. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  24. 2020Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformerscited by 57
  25. 2023An Open Dataset and Model for Language Identificationcited by 49
  26. 2020Inducing Language-Agnostic Multilingual Representationscited by 48
  27. 2023Explaining How Transformers Use Context to Build Predictionscited by 42
  28. 2022Mini-Model Adaptation: Efficiently Extending Pretrained Models to New Languages via Aligned Shallow Trainingcited by 24
  29. 2024Discovering Low-rank Subspaces for Language-agnostic Multilingual Representationscited by 18
  30. 2023Morphosyntactic probing of multilingual BERT modelscited by 8
  31. 2022Adapting BigScience Multilingual Model to Unseen Languagescited by 6
  32. 2024The Impact of Language Adapters in Cross-Lingual Transfer for NLUcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 6; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  2. 2025Multilingual Routing in Mixture-of-Expertscited by 25
  3. 2024Layer Swapping for Zero-Shot Cross-Lingual Transfer in Large Language Modelscited by 20
  4. 2025Language-Specific Latent Process Hinders Cross-Lingual Performancecited by 20
  5. 2025Localized Cultural Knowledge is Conserved and Controllable in Large Language Modelscited by 19
  6. 2025Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Aheadcited by 15
  7. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  8. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 9
  9. 2024The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experimentscited by 8
  10. 2026Translation as a Scalable Proxy for Multilingual Evaluationcited by 6
  11. 2024Soft Language Prompts for Language Transfercited by 5
  12. 2024UniBridge: A Unified Approach to Cross-Lingual Transfer Learning for Low-Resource Languagescited by 4
4 more
  1. 2024Exploring Cross-lingual Latent Transplantation: Mutual Opportunities and Open Challengescited by 1
  2. 2026Leveraging Routing Dynamics in Mixture-of-Experts Models for Efficient Language Adaptationcited by 1
  3. 2024Adapter-based Approaches to Knowledge-enhanced Language Models -- A Surveycited by 0
  4. 2026Can LLMs Cook Jamaican Couscous? A Study of Cultural Novelty in Recipe Generationcited by 0