Evaluating Large Vision Language Models on Bangla Medical Visual Question Answering

Key Contributions

The BanglaMedVQA Dataset

BanglaMedVQA is built on two widely used biomedical sources — ChestX-ray8 and MedICaT — pairing structured chest-radiograph metadata with a broader range of imaging modalities and anatomical regions. For every image, English QA pairs are generated from its metadata using gemini-2.5-flash with few-shot prompting, then translated into Bangla with the same model. The result is 7,000 clinically grounded image–question–answer pairs, balanced across healthy and abnormal cases and split evenly across five question categories.

A three-stage curation pipeline

A blind validation study by two compensated medical specialists confirmed the pipeline’s reliability: generated QA pairs reached a 97% acceptance rate with strong inter-rater agreement (Cohen’s κ = 0.89), covering both translation quality and anatomical-position correctness.

Dataset distribution

Evaluation Setup

We benchmark nine LVLMs — closed-source (Gemini 2.5 Flash, GPT-4.1 Mini, Claude Sonnet 4), open-source general-purpose (LLaMA-3.2V, Gemma-3, Qwen2.5-VL, LLaVA-1.5), and medical-domain (Med-LLaVA, Med-Gemma) — under three strategies. Open-source models are additionally LoRA fine-tuned (rank 16, α 32, 2 epochs) on a single NVIDIA A100. Every answer is scored with three complementary metrics.

Quantitative Benchmarking Results

Contemporary vision-language models like Gemini 2.5 Flash and Claude Sonnet 4 answer general imaging questions reasonably well in English, but accuracy drops sharply once the same questions are posed in Bangla. LoRA fine-tuning dramatically closes this gap on open-source backbones such as Qwen2.5-VL and Med-Gemma 4B, lifting the best overall LAVE average to nearly ~76%. Use the tabs below to switch between the Bangla and English benchmarks.

Key findings

Fine-Grained Performance Analysis

Error Analysis

Models generally handle straightforward questions — identifying the imaging modality or the organ involved — but struggle with specialized diagnostic reasoning. Errors are most frequent on questions requiring precise localization or interpretation of pathological findings, where models confuse nearby anatomical regions or misclassify related conditions such as cysts and edema. Many responses are only partially correct: they capture a general abnormality while missing the precise clinical detail. These failures point to gaps in visual–textual grounding and domain understanding — gaps that LoRA fine-tuning noticeably, but not fully, narrows.

Conclusion

We present the first Medical Visual Question Answering benchmark for Bangla, addressing a significant gap in low-resource, multimodal evaluation. Across a systematic study of nine open- and closed-source LVLMs, top models such as Gemini, GPT-4.1 Mini, and Claude Sonnet 4 perform reasonably on general questions but fail on specialized diagnostic tasks. LoRA fine-tuning delivers substantial gains — enabling open-source models like Qwen2.5-VL and Med-Gemma 4B to surpass closed-source baselines — yet performance on the Condition and Position categories remains limited, showing that current VLMs still lack fine-grained medical reasoning. BanglaMedVQA offers a clinically validated foundation to drive future progress.

Intended use. Models trained on this dataset are for research only and must not be used for clinical decision-making. All images and annotations come from publicly available sources with no personally identifiable information.

BibTeX citation

@inproceedings{ahmed-etal-2026-evaluating,
    title = "Evaluating Large Vision Language Models on {B}angla Medical Visual Question Answering",
    author = "Ahmed, Rafid  and
      Tahmid, Intesar  and
      Hossain, Mir Sazzat  and
      Tomal, Tasnimul Hossain  and
      Jawad, Md Mahir  and
      Uddin, Anam Borhan  and
      Fahim, Md  and
      Bhuiyan, Md Farhad Alam",
    editor = "Liakata, Maria  and
      Moreira, Viviane P.  and
      Zhang, Jiajun  and
      Jurgens, David",
    booktitle = "Findings of the {A}ssociation for {C}omputational {L}inguistics: {ACL} 2026",
    month = jul,
    year = "2026",
    address = "San Diego, California, United States",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2026.findings-acl.1862/",
    pages = "37362--37378",
    ISBN = "979-8-89176-395-1",
    abstract = "Recent advancements in Large Language Models (LLMs) and Large Vision Language Models (LVLMs) have enabled general-purpose systems to demonstrate promising capabilities in complex reasoning tasks, including those in the medical domain. However, their evaluation has predominantly focused on high-resource languages, leaving low-resource contexts like Bangla underexplored. To address this gap, we introduce BanglaMedVQA, a multilingual Medical Visual Question Answering (VQA) dataset comprising clinically validated image{--}question{--}answer pairs, along with a comprehensive evaluation of current LVLMs on this resource. We rigorously evaluate nine state-of-the-art LVLMs using zero-shot, Chain-of-Thought (CoT), and LoRA fine-tuning strategies. Our results reveal a clear performance disparity: models perform well on generalized visual tasks but struggle with fine-grained diagnostic reasoning, achieving surprisingly low accuracy in specialized categories. While fine-tuning significantly improves overall accuracy, especially for Qwen2.5-VL and MedGemma 4B, limitations in specialized medical reasoning persist. Our work provides a foundation for future research in Bangla medical VQA."
}