# KrishokChat: A Provenance-Traceable Multi-Task Bengali Agricultural Benchmark with Safety-Critical Chemical Advisory

arXiv: https://arxiv.org/abs/2606.29243 · HTML: https://mlsanity.com/paper/2606.29243/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 28 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance](https://mlsanity.com/paper/2105.07624/) (2021)
- [PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference](https://mlsanity.com/paper/2406.15513/) (2024)
- [Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails](https://mlsanity.com/paper/2501.09004/) (2025)
- [BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla](https://mlsanity.com/paper/2101.00204/) (2021)
- [Learning to Ask Good Questions: Ranking Clarification Questions using Neural Expected Value of Perfect Information](https://mlsanity.com/paper/1805.04655/) (2018)
- [FairLex: A Multilingual Benchmark for Evaluating Fairness in Legal Text Processing](https://mlsanity.com/paper/2203.07228/) (2022)
- [BanglaNLG and BanglaT5: Benchmarks and Resources for Evaluating Low-Resource Natural Language Generation in Bangla](https://mlsanity.com/paper/2205.11081/) (2022)
- [Diversity Measurement and Subset Selection for Instruction Tuning Datasets](https://mlsanity.com/paper/2402.02318/) (2024)
- [Farmer.Chat: Scaling AI-Powered Agricultural Services for Smallholder Farmers](https://mlsanity.com/paper/2409.08916/) (2024)
- [Vashantor: A Large-scale Multilingual Benchmark Dataset for Automated Translation of Bangla Regional Dialects to Bangla Language](https://mlsanity.com/paper/2311.11142/) (2023)
- [Table Question Answering for Low-resourced Indic Languages](https://mlsanity.com/paper/2410.03576/) (2024)

## Built on by (most cited first, 1 of 1)

- [Where Does Retrieval Fail? Evaluating RAG Architectures for Agricultural Advisory](https://mlsanity.com/paper/2608.14886/) (2026)

## Abstract (arXiv)

We introduce KrishokChat, an 85,979-instance Bengali agricultural benchmark built from 284 government publications, 13 institutions, and six regional dialects. The benchmark comprises four tracks: General Knowledge QA, Treatment QA, Safety Refusal and Re-query, and Table QA. It also includes a 1,000-query Real-World Farmer Benchmark collected independently from field interviews to measure transfer to authentic farmer language. Every extracted instance retains provenance at the citation level. Treatment QA also provides a structured chemical-trace array for dosage-level auditability. We evaluated five zero-shot baselines and one fine-tuned model. Closed-book knowledge proves insufficient regardless of model scale. Oracle evidence narrows the gap, but leaves a persistent floor of 4.05 to 7.00% of chemical hallucinations. Fine-tuning on KrishokChat substantially outperforms the strongest zero-shot baseline on General QA Token F1. However, structured table reasoning and farmer-language transfer remain largely unsolved. An exploratory analysis further reveals that fine-tuning reduces refusal behavior on safety-critical queries, indicating that additional safety alignment is required beyond supervised fine-tuning. We release KrishokChat as a traceable benchmark and audit resource to support grounded and safety-aware agricultural language modeling for Bengali.