mlsanityData to 14 Oct 2026

Performance Evaluation of Large Language Models in Bangla Consumer Health Query Summarization

arXiv 2505.05070 ↗First posted 8 May 2025cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

Consumer Health Queries (CHQs) in Bengali (Bangla), a low-resource language, often contain extraneous details, complicating efficient medical responses. This study investigates the zero-shot performance of nine advanced large language models (LLMs): GPT-3.5-Turbo, GPT-4, Claude-3.5-Sonnet, Llama3-70b-Instruct, Mixtral-8x22b-Instruct, Gemini-1.5-Pro, Qwen2-72b-Instruct, Gemma-2-27b, and Athene-70B, in summarizing Bangla CHQs. Using the BanglaCHQ-Summ dataset comprising 2,350 annotated query-summary pairs, we benchmarked these LLMs using ROUGE metrics against Bangla T5, a fine-tuned state-of-the-art model. Mixtral-8x22b-Instruct emerged as the top performing model in ROUGE-1 and ROUGE-L, while Bangla T5 excelled in ROUGE-2. The results demonstrate that zero-shot LLMs can rival fine-tuned models, achieving high-quality summaries even without task-specific training. This work underscores the potential of LLMs in addressing challenges in low-resource languages, providing scalable solutions for healthcare query summarization.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 21 of its 31 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

21 of 31 references.

References · 21

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2024Qwen2 Technical Reportcited by 2,120
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2024Mixtral of Expertscited by 1,823
  7. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  8. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 798
  9. 2023A Comprehensive Capability Analysis of GPT-3 and GPT-3.5 Series Modelscited by 246
  10. 2021BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Banglacited by 125
  11. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 92
  12. 2024Towards Building Multilingual Language Model for Medicinecited by 92
9 more
  1. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  2. 2017Text Summarization Techniques: A Brief Surveycited by 59
  3. 2022BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Banglacited by 42
  4. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  5. 2023BenLLMEval: A Comprehensive Evaluation into the Potentials and Pitfalls of Large Language Models on Bengali NLPcited by 23
  6. 2023Evaluation of ChatGPT on Biomedical Tasks: A Zero-Shot Comparison with Fine-Tuned Generative Transformerscited by 22
  7. 2024Assessing LLMs for Zero-shot Abstractive Summarization Through the Lens of Relevance Paraphrasingcited by 11
  8. 2024OlympicArena Medal Ranks: Who Is the Most Intelligent AI So Far?cited by 10
  9. 2024Comparative Analysis of Open-Source Language Models in Summarizing Medical Text Datacited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Preemptive Hallucination Reduction: An Input-Level Approach for Multimodal Language Modelcited by 2
  2. 2025From Chat to Checkup: Can Large Language Models Assist in Diabetes Prediction?cited by 1
  3. 2025BanglaMedQA and BanglaMMedBench: Evaluating Retrieval-Augmented Generation Strategies for Bangla Biomedical Question Answeringcited by 1
  4. 2026Few-Shot Learning Pipeline for Monkeypox Skin Disease Classification Using CNN Feature Extractorscited by 0
  5. 2026Beyond Accuracy: A Qualitative Analysis of Vision-Language Models for Hate Speech Detection in Memescited by 0
  6. 2026Language Specificity vs. Domain Diversity: Benchmarking Transformers for Bangla Medical NERcited by 0