mlsanityData to 14 Oct 2026

Symbol tuning improves in-context learning in language models

arXiv 2305.08298 ↗First posted 15 May 2023cs.CLcited by 75

Not on the map. Why?

Abstract (arXiv)

We present symbol tuning - finetuning language models on in-context input-label pairs where natural language labels (e.g., "positive/negative sentiment") are replaced with arbitrary symbols (e.g., "foo/bar"). Symbol tuning leverages the intuition that when a model cannot use instructions or natural language labels to figure out a task, it must instead do so by learning the input-label mappings. We experiment with symbol tuning across Flan-PaLM models up to 540B parameters and observe benefits across various settings. First, symbol tuning boosts performance on unseen in-context learning tasks and is much more robust to underspecified prompts, such as those without instructions or without natural language labels. Second, symbol-tuned models are much stronger at algorithmic reasoning tasks, with up to 18.2% better performance on the List Functions benchmark and up to 15.3% better performance on the Simple Turing Concepts benchmark. Finally, symbol-tuned models show large improvements in following flipped-labels presented in-context, meaning that they are more capable of using in-context information to override prior semantic knowledge.

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Builds on 43 of its 71 references that are in the map. First posted on arXiv 15 May 2023.

Builds on

43 of 71 references.

References · 43

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  12. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
31 more
  1. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  5. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  6. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  8. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  10. 2022Transformers learn in-context by gradient descentcited by 732
  11. 2022What Can Transformers Learn In-Context? A Case Study of Simple Function Classescited by 715
  12. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 699
  13. 2022What learning algorithm is in-context learning? Investigations with linear modelscited by 684
  14. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 606
  15. 2021Datasets: A Community Library for Natural Language Processingcited by 530
  16. 2021MetaICL: Learning to Learn In Contextcited by 454
  17. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 402
  18. 2023Larger language models do in-context learning differentlycited by 394
  19. 2019Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAMLcited by 359
  20. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 326
  21. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  22. 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matterscited by 222
  23. 2019SemEval-2017 Task 4: Sentiment Analysis in Twittercited by 210
  24. 2019SemEval-2019 Task 6: Identifying and Categorizing Offensive Language in Social Media (OffensEval)cited by 207
  25. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  26. 2021CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLPcited by 142
  27. 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tangocited by 128
  28. 2019Meta-Learning without Memorizationcited by 92
  29. 2020Meta-Learning Requires Meta-Augmentationcited by 57
  30. 2021RAFT: A Real-World Few-Shot Text Classification Benchmarkcited by 44
  31. 2021Symbolic Behaviour in Artificial Intelligencecited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 32; 2025: 19; 2026: 5.

75 papers.

Sort:

By area:

  1. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  3. 2023Simple synthetic data reduces sycophancy in large language modelscited by 187
  4. 2024Best Practices and Lessons Learned on Synthetic Datacited by 106
  5. 2024RAG-Driver: Generalisable Driving Explanations with Retrieval-Augmented In-Context Learning in Multi-Modal Large Language Modelcited by 92
  6. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  7. 2023In-Context Learning with Iterative Demonstration Selectioncited by 74
  8. 2023The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AIcited by 73
  9. 2024Inference Scaling for Long-Context Retrieval Augmented Generationcited by 65
  10. 2023QUAR-VLA: Vision-Language-Action Model for Quadruped Robotscited by 56
  11. 2024What Makes Multimodal In-Context Learning Work?cited by 56
  12. 2024Universal Vulnerabilities in Large Language Models: Backdoor Attacks for In-context Learningcited by 53
38 more
  1. 2023Data-Juicer: A One-Stop Data Processing System for Large Language Modelscited by 52
  2. 2024Why Larger Language Models Do In-context Learning Differently?cited by 45
  3. 2024Do Large Language Models Have Compositional Ability? An Investigation into Limitations and Scalabilitycited by 40
  4. 2025Advancing Expert Specialization for Better MoEcited by 39
  5. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
  6. 2023Teaching Language Models to Hallucinate Less with Synthetic Taskscited by 31
  7. 2023Instruction-following Evaluation through Verbalizer Manipulationcited by 30
  8. 2024DARG: Dynamic Evaluation of Large Language Models via Adaptive Reasoning Graphcited by 26
  9. 2024ADELIE: Aligning Large Language Models on Information Extractioncited by 24
  10. 2023FLM-101B: An Open LLM and How to Train It with 100K Budgetcited by 22
  11. 2024Bypassing the Exponential Dependency: Looped Transformers Efficiently Learn In-context by Multi-step Gradient Descentcited by 22
  12. 2024On the Noise Robustness of In-Context Learning for Text Generationcited by 16
  13. 2023Confronting LLMs with Traditional ML: Rethinking the Fairness of Large Language Models in Tabular Classificationscited by 15
  14. 2024Interactive Evolution: A Neural-Symbolic Self-Training Framework For Large Language Modelscited by 14
  15. 2024Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Taskscited by 13
  16. 2023Large Language Models are Miscalibrated In-Context Learnerscited by 12
  17. 2024FabGPT: An Efficient Large Multimodal Model for Complex Wafer Defect Knowledge Queriescited by 12
  18. 2023Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Modelscited by 11
  19. 2024Do LLMs Really Adapt to Domains? An Ontology Learning Perspectivecited by 11
  20. 2024Mitigating Copy Bias in In-Context Learning through Neuron Pruningcited by 11
  21. 2024Understanding Synthetic Context Extension via Retrieval Headscited by 10
  22. 2024SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimizationcited by 10
  23. 2025Theoretical Guarantees for High Order Trajectory Refinement in Generative Flowscited by 10
  24. 2023NICE: CVPR 2023 Challenge on Zero-shot Image Captioningcited by 9
  25. 2024Token-based Decision Criteria Are Suboptimal in In-context Learningcited by 9
  26. 2025Fast RoPE Attention: Combining the Polynomial Method and Fast Fourier Transformcited by 9
  27. 2025The Few-shot Dilemma: Over-prompting Large Language Modelscited by 8
  28. 2024Leveraging Professional Radiologists' Expertise to Enhance LLMs' Evaluation for Radiology Reportscited by 7
  29. 2024FsPONER: Few-shot Prompt Optimization for Named Entity Recognition in Domain-specific Scenarioscited by 7
  30. 2023JsonTuning: Towards Generalizable, Robust, and Controllable Instruction Tuningcited by 6
  31. 2023AutoParLLM: GNN-guided Context Generation for Zero-Shot Code Parallelization using LLMscited by 5
  32. 2023ICL Markup: Structuring In-Context Learning using Soft-Token Tagscited by 5
  33. 2023Beyond Output Matching: Bidirectional Alignment for Enhanced In-Context Learningcited by 5
  34. 2024PromptRefine: Enhancing Few-Shot Performance on Low-Resource Indic Languages with Example Selection from Related Example Bankscited by 5
  35. 2023Explanation-aware Soft Ensemble Empowers Large Language Model In-context Learningcited by 4
  36. 2024Speak It Out: Solving Symbol-Related Problems with Symbol-to-Language Conversion for Language Modelscited by 4
  37. 2024NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learningcited by 4
  38. 2024Investigating the Effectiveness of HyperTuning via Gistingcited by 4

50 most cited of 75 All 75 (JSON)