Continual-learning for Modelling Low-Resource Languages from Large Language Models
Not on the map. Why?
Abstract (arXiv)
Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 33 of its 56 references that are in the map. First posted on arXiv 9 Jan 2026.
Builds on
33 of 56 references.
References · 33
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
- 2016Learning without Forgettingcited by 2,496
- 2016Understanding intermediate layers using linear classifier probescited by 1,353
- 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
- 2018Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigencecited by 742
- 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
- 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
- 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
- 2020AdapterHub: A Framework for Adapting Transformerscited by 444
- 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
21 more
- 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 318
- 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
- 2023NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenariocited by 262
- 2018A Comprehensive Survey of Deep Learning for Image Captioningcited by 214
- 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
- 2020MTOP: A Comprehensive Multilingual Task-Oriented Semantic Parsing Benchmarkcited by 158
- 2023IndicTrans2: Towards High-Quality and Accessible Machine Translation Models for all 22 Scheduled Indian Languagescited by 152
- 2021From Show to Tell: A Survey on Deep Learning-based Image Captioningcited by 138
- 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
- 2018Don't forget, there is more than forgetting: new metrics for Continual Learningcited by 115
- 2021NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Mediacited by 107
- 2021xGQA: Cross-Lingual Visual Question Answeringcited by 72
- 2022Overcoming Catastrophic Forgetting in Incremental Object Detection via Elastic Response Distillationcited by 72
- 2021UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-trainingcited by 64
- 2023mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMscited by 41
- 2024Adaptive Memory Replay for Continual Learningcited by 22
- 2019Overcoming Catastrophic Forgetting by Generative Regularizationcited by 13
- 2022Large-scale Bilingual Language-Image Contrastive Learningcited by 12
- 2023PAXQA: Generating Cross-lingual Question Answering Examples at Training Scalecited by 10
- 2024Emerging Practices for Large Multimodal Model (LMM) Assistance for People with Visual Impairments: Implications for Designcited by 10
- 2022Cross-lingual Lifelong Learningcited by 9
Built on by
Citing papers per year: 2026: 1.
1 paper.