Simple and Effective Input Reformulations for Translation
Not on the map. Why?
Abstract (arXiv)
Foundation language models learn from their finetuning input context in different ways. In this paper, we reformulate inputs during finetuning for challenging translation tasks, leveraging model strengths from pretraining in novel ways to improve downstream performance. These reformulations are simple data level modifications, require no additional collection of training data or modification of data at inference time. They can be applied either on single language pair translation tasks or massively multilingual translation tasks. Experiments with these techniques demonstrate significant performance improvements up to 3.5 chrF++ on the Flores200 translation benchmark. We hope our research accessibly improves finetuning data efficiency, enabling more effective training to scalably improve state-of-the-art performance. Our code is released hrefhttps://github.com/bri25yu/LanguageModelExperimentationhere.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 51 of its 65 references that are in the map. First posted on arXiv 12 Nov 2023.
Builds on
51 of 65 references.
References · 51
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
- 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
- 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
- 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
- 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
- 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
38 more
- 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
- 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
- 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
- 2019Language Models as Knowledge Bases?cited by 1,936
- 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
- 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
- 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
- 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
- 2021Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivitycited by 1,135
- 2019Cross-lingual Language Model Pretrainingcited by 1,090
- 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
- 2019How Can We Know What Language Models Know?cited by 959
- 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
- 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
- 2015Improving Neural Machine Translation Models with Monolingual Datacited by 788
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
- 2018Learning to Reweight Examples for Robust Deep Learningcited by 675
- 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 670
- 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 560
- 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
- 2019ERNIE: Enhanced Representation through Knowledge Integrationcited by 302
- 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
- 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
- 2022OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalizationcited by 236
- 2021WARP: Word-level Adversarial ReProgrammingcited by 221
- 2019Competence-based Curriculum Learning for Neural Machine Translationcited by 202
- 2021KnowPrompt: Knowledge-aware Prompt-tuning with Synergistic Optimization for Relation Extractioncited by 180
- 2017Data Augmentation for Low-Resource Neural Machine Translationcited by 139
- 2021Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Modelscited by 139
- 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
- 2018Meta-Learning for Low-Resource Neural Machine Translationcited by 105
- 2020Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classificationcited by 98
- 2021BERTese: Learning to Speak to BERTcited by 82
- 2017Curriculum Learning and Minibatch Bucketing in Neural Machine Translationcited by 63
- 2021PADA: Example-based Prompt Learning for on-the-fly Adaptation to Unseen Domainscited by 61
- 2019Curriculum Learning for Domain Adaptation in Neural Machine Translationcited by 59
- 2018An Empirical Exploration of Curriculum Learning for Neural Machine Translationcited by 58
50 most cited of 51 All 51 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.