mlsanityData to 14 Oct 2026

Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation

arXiv 2606.18389 ↗First posted 16 Jun 2026cs.CLcited by 0

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

also in 1 more
Abstract (arXiv)

Large language models (LLMs) have become an effective tool for synthetic data generation, including for low-resource languages, where generated data can improve downstream task performance. Current best-performing approaches typically rely on few-shot prompting with target-language examples, which increases inference costs and may reduce diversity through lexical anchoring. In this work, we investigate activation steering as an alternative for low-resource synthetic data generation. We study two steering strategies: Language Steering, which targets the linguistic identity of a language, and Quality Steering, which captures well-formedness by contrasting human-written and backtranslated text representations. We evaluate these methods across four open-source LLMs, multiple layers, and 11 typologically diverse languages by generating sentiment and topic classification data and finetuning smaller classifiers. Steering is applied in both zero-shot and few-shot prompting settings and compared against non-steered counterparts. Our results show that steering on early layers consistently improves the diversity of generated data while often yielding stronger downstream model performance, particularly for low-resource languages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 39 of its 66 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

39 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2026Better as Generators Than Classifiers: Leveraging LLMs and Synthetic Data for Low-Resource Multilingual Classification22 Jan
  2. 2025RoSE: Round-robin Synthetic Data Evaluation for Selecting LLM Generators without Human Test Sets7 Oct

Other references · 37

  1. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  8. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  9. 2023Steering Language Models With Activation Engineeringcited by 922
  10. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  11. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  12. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
25 more
  1. 2021A Survey of Data Augmentation Approaches for NLPcited by 409
  2. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 218
  3. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  4. 2023SIB-200: A Simple, Inclusive, and Big Evaluation Dataset for Topic Classification in 200+ Languages and Dialectscited by 152
  5. 2024How do Large Language Models Handle Multilingualism?cited by 150
  6. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  7. 2023ChatGPT to Replace Crowdsourcing of Paraphrases for Intent Classification: Higher Diversity and Comparable Model Robustnesscited by 41
  8. 2022Memory-efficient NLLB-200: Language-specific Expert Pruning of a Massively Multilingual Machine Translation Modelcited by 36
  9. 2023ZeroShotDataAug: Generating and Augmenting Training Data with ChatGPTcited by 32
  10. 2020Diversity, Density, and Homogeneity: Quantitative Characteristic Metrics for Text Collectionscited by 22
  11. 2025BOUQuET: dataset, Benchmark and Open initiative for Universal Quality Evaluation in Translationcited by 22
  12. 2023Using GPT-4 to Augment Unbalanced Data for Automatic Scoringcited by 20
  13. 2024Effects of diversity incentives on sample diversity and downstream model performance in LLM-based text augmentationcited by 17
  14. 2025Language Arithmetics: Towards Systematic Language Neuron Identification and Manipulationcited by 17
  15. 2024Adapting Multilingual LLMs to Low-Resource Languages with Knowledge Graphs via Adapterscited by 15
  16. 2025Causal Language Control in Multilingual Transformers via Sparse Feature Steeringcited by 12
  17. 2026CLaS-Bench: A Cross-Lingual Alignment and Steering Benchmarkcited by 12
  18. 2025Beyond Translation: LLM-Based Data Generation for Multilingual Fact-Checkingcited by 8
  19. 2026From Weights to Activations: Is Steering the Next Frontier of Adaptation?cited by 8
  20. 2024GrEmLIn: A Repository of Green Baseline Embeddings for 87 Low-Resource Languages Injected with Multilingual Graph Knowledgecited by 7
  21. 2024Use Random Selection for Now: Investigation of Few-Shot Selection Strategies in LLM-based Text Augmentation for Classificationcited by 7
  22. 2026Compositional Steering of Large Language Models with Steering Tokenscited by 5
  23. 2026Multilingual Steering by Design: Multilingual Sparse Autoencoders and Principled Layer Selectioncited by 4
  24. 2026LangFIR: Discovering Sparse Language-Specific Features from Monolingual Data for Language Steeringcited by 1
  25. 2026DFKI-MLT at SemEval-2026 TASK 7: Steering Multilingual Models Towards Cultural Knowledgecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025In-Distribution Steering: Balancing Control and Coherence in Language Model Generation
  2. 2023Steering Language Generation: Harnessing Contrastive Expert Guidance and Negative Prompting for Coherent and Diverse Synthetic Data Generation
  3. 2025Steering When Necessary: Flexible Steering Large Language Models with Backtracking
  4. 2026COLD-Steer: Steering Large Language Models via In-Context One-step Learning Dynamics
  5. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  6. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
  7. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  8. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations