mlsanityData to 14 Oct 2026

SASFT: Sparse Autoencoder-guided Supervised Finetuning to Mitigate Unexpected Code-Switching in LLMs

arXiv 2507.14894 ↗First posted 20 Jul 2025cs.CLcited by 2

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Large Language Models (LLMs) have impressive multilingual capabilities, but they suffer from unexpected code-switching, also known as language mixing, which involves switching to unexpected languages in the model response. This problem leads to poor readability and degrades the usability of model responses. However, existing work on this issue lacks a mechanistic analysis and shows limited effectiveness. In this paper, we first provide an in-depth analysis of unexpected code-switching using sparse autoencoders and find that when LLMs switch to a language, the features of that language exhibit excessive pre-activation values. Based on our findings, we propose Sparse Autoencoder-guided Supervised Finetuning (SASFT), which teaches LLMs to maintain appropriate pre-activation values of specific language features during training. Experiments on five models across three languages demonstrate that SASFT consistently reduces unexpected code-switching by more than 50% compared to standard supervised fine-tuning, with complete elimination in one case. Moreover, SASFT maintains or even improves the models' performance on six multilingual benchmarks, showing its effectiveness in addressing code-switching while preserving multilingual capabilities. The code and data are available at https://github.com/Aatrox103/SASFT.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 43 of its 57 references that are in the map. First posted on arXiv 20 Jul 2025.

Builds on

43 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Unveiling Language-Specific Features in Large Language Models via Sparse Autoencoders8 May
  2. 2025Route Sparse Autoencoder to Interpret Large Language Models11 Mar
  3. 2024Can sparse autoencoders be used to decompose and interpret steering vectors?13 Nov
  4. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  5. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  6. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 37

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  10. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
25 more
  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  2. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 629
  3. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
  4. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 530
  5. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  6. 2024Do I Know This Entity? Knowledge Awareness and Hallucinations in Language Modelscited by 115
  7. 2024Multi-IF: Benchmarking LLMs on Multi-Turn and Multilingual Instructions Followingcited by 102
  8. 2024HumanEval-XL: A Multilingual Code Generation Benchmark for Cross-lingual Natural Language Generalizationcited by 86
  9. 2023Multilingual Large Language Models Are Not (Yet) Code-Switcherscited by 65
  10. 2023PolyLM: An Open Source Polyglot Large Language Modelcited by 64
  11. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 51
  12. 2024Multilingual Large Language Model: A Survey of Resources, Taxonomy and Frontierscited by 51
  13. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  14. 2024A Survey on Large Language Models with Multilingualism: Recent Advances and New Frontierscited by 40
  15. 2022The Decades Progress on Code-Switching Research in NLP: A Systematic Survey on Trends and Challengescited by 35
  16. 2024P-MMEval: A Parallel Multilingual Multitask Benchmark for Consistent Evaluation of LLMscited by 29
  17. 2024Code-Switching Curriculum Learning for Multilingual Transfer in LLMscited by 27
  18. 2025Language Mixing in Reasoning Language Models: Patterns, Impact, and Internal Causescited by 21
  19. 2024PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignmentcited by 20
  20. 2025Investigating and Scaling up Code-Switching for Multilingual Language Model Pre-Trainingcited by 16
  21. 2023GlotScript: A Resource and Tool for Low Resource Writing System Identificationcited by 15
  22. 2024MINERS: Multilingual Language Models as Semantic Retrieverscited by 11
  23. 2025Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Modelscited by 8
  24. 2024Linguistics Theory Meets LLM: Code-Switched Text Generation via Equivalence Constrained Large Language Modelscited by 6
  25. 2024Code-Switched Language Identification is Harder Than You Thinkcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Qwen-Scope: Turning Sparse Features into Development Tools for Large Language Modelscited by 21
  2. 2026Code Mixologist : A Practitioner's Guide to Building Code-Mixed LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models
  2. 2025Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models
  3. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  4. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
  5. 2024Scaling Sparse Fine-Tuning to Large Language Models
  6. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steering
  7. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability
  8. 2026When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs