mlsanityData to 14 Oct 2026

Beyond Transfer Accuracy: Mechanism-Guided Controlled Adaptation for Low-Resource Languages

arXiv 2601.08146 ↗First posted 13 Jan 2026cs.CL cs.AI cs.LGcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Existing circuit discovery methods rely on templated tasks with clean counterfactuals, limiting their use on diverse natural text. We adapt Contextual Decomposition for Transformers (CD-T) for unstructured settings via label-balanced activation means and task-directional relevance scoring, enabling counterfactual-free circuit discovery. We leverage the discovered circuits for Circuit-Targeted Supervised Fine-Tuning (CT-SFT), restricting parameter updates to task-relevant heads and LayerNorm. Experiments on NusaX cross-lingual sentiment transfer show that CT-SFT is highly competitive for low-resource adaptation. While non-circuit sparse updates and full fine-tuning sometimes match target accuracy through capacity recruitment, CT-SFT most consistently avoids catastrophic forgetting, preserving source-language and related-task performance. Extensions to XNLI support the source-retention and intervention findings on a harder task and two model families, showing that circuit-targeted adaptation provides a more controlled, intervention-supported alternative to global fine-tuning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 31 of its 43 references that are in the map. First posted on arXiv 13 Jan 2026.

Builds on

31 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gates15 May
  2. 2025Fine-Tuning is Subgraph Search: A New Lens on Learning Dynamics10 Feb
  3. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identification7 Feb
  4. 2024Efficient Automated Circuit Discovery in Transformers using Contextual Decomposition1 Jul
  5. 2024Mechanistic Interpretability for AI Safety -- A Review22 Apr
  6. 2023Attribution Patching Outperforms Automated Circuit Discovery16 Oct
  7. 2023Circuit Component Reuse Across Tasks in Transformer Language Models12 Oct
  8. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr
  9. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small1 Nov

Other references · 22

  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  2. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 1,006
  3. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  4. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  5. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  6. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  7. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 255
  8. 2021Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Modelscited by 222
  9. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  10. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 149
  11. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  12. 2020IndoNLU: Benchmark and Resources for Evaluating Indonesian Natural Language Understandingcited by 109
10 more
  1. 2022NusaX: Multilingual Parallel Sentiment Dataset for 10 Indonesian Local Languagescited by 68
  2. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  3. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  4. 2024Explorations of Self-Repair in Language Modelscited by 39
  5. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  6. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 28
  7. 2019Sparse Transfer Learning via Winning Lottery Ticketscited by 27
  8. 2025Towards Understanding Fine-Tuning Mechanisms of LLMs via Circuit Analysiscited by 20
  9. 2025LoraxBench: A Multitask, Multilingual Benchmark Suite for 20 Indonesian Languagescited by 6
  10. 2025NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scriptscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Attribution-Guided Pruning for Insight and Control: Circuit Discovery and Targeted Correction in Small-scale LLMs
  2. 2026Aletheia: Gradient-Guided Layer Selection for Efficient LoRA Fine-Tuning Across Architectures
  3. 2026Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training
  4. 2025Balancing Fidelity and Plasticity: Aligning Mixed-Precision Fine-Tuning with Linguistic Hierarchies
  5. 2026Where to Adapt Matters: Layer-Selective Fine-Tuning for Capability Retention
  6. 2026Where Does Robustness Live? Neuron-Guided Adaptation for Retrieval-Augmented Language Models
  7. 2026GIFT: Guided Fine-Tuning and Transfer for Enhancing Instruction-Tuned Language Models
  8. 2025LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation