PatientQA: A Question Answering Benchmark for Patient Diagnosis

Blog · Dec 2024 · Medical QA evaluation

Introduction

Medical evaluation has largely focused on multiple-choice questions, short answers, and recognition of information in medical images. However, clinical practice also requires generating a detailed diagnosis from a patient’s background and interpreting images in that context. PatientQA is designed as a complementary benchmark for these patient-related question-answering abilities, covering both text-only and multimodal evaluation.

Tasks for LLM/MLLM

PatientQA contains two complementary tasks. The OpenQA subset contains 487 questions in two parts: medical history collection and patient-summary diagnosis. Models produce longer diagnostic responses, which are evaluated with a step-by-step comparison against reference answers. The multimodal MCQ subset contains approximately 665 questions collected from medical examination material; each question requires an LLM or MLLM to use the patient context and, where provided, a medical image to select the correct answer.

The benchmark therefore tests whether LLMs can generate comprehensive patient-related medical text and whether MLLMs can integrate visual information into patient diagnosis.

How PatientQA differs from earlier datasets

Most earlier medical datasets emphasize short answers, image recognition, or knowledge recall. PatientQA combines long-form patient diagnosis with multimodal questions that require medical-image interpretation in a patient context. The table summarizes this difference against representative medical QA benchmarks.

Across the text-based benchmarks we surveyed, most questions are multiple-choice or short-answer questions. In multimodal benchmarks, questions more often test image recognition, organ identification, or general knowledge than direct diagnosis from patient context.

The dataset statistics show how the two PatientQA tasks are organized: the MCQ subset is grouped by medical specialty, while the OpenQA subset is grouped by medical system. These distributions reflect the source examination material and the clinical contexts represented in the benchmark.

Settings

PatientQA has two parts: roughly 665 multimodal multiple-choice questions and 487 open-ended questions covering medical history collection and patient-summary diagnosis. For OpenQA, a step-by-step judge compares an answer with a reference before producing a total score. For multimodal MCQ, the evaluation focuses on patient-centered diagnosis rather than generic image recognition.

The experiments compare zero-shot, one-shot, chain-of-thought, and knowledge-augmented settings. For text-only OpenQA, we evaluate Qwen2.5-7B-Instruct, ChatGLM-2-6B, Llama-3.1-8B-Instruct, Vicuna-7B-v1.5, GPT-3.5-Turbo, GPT-4o-mini, and Gemini-1.5-Flash. For multimodal MCQ, we evaluate GPT-4o-mini together with open-source MLLMs including Qwen2-VL, LLaVA-1.5, LLaVA-Med, InstructBLIP, and Llama-3.2-Vision. This mix covers both open-source models at roughly the 7B–11B scale and API-based models.

For MCQ, the evaluation extracts the candidate letter from the model output. For OpenQA, a first judge pass compares the response with the reference step by step, and a second pass extracts the total score. The knowledge-augmented setting supplies the diagnostic cheat sheets included with the dataset.

Results

The benchmark-level observation is clear: most models perform only marginally above random selection on the multimodal MCQ setting, and even the strongest systems generally remain below the 60% passing threshold on the patient-summary diagnosis task. On the long-answer task, the proposed semantic score ranked GPT-4o-mini at 52.04/100, while BLEU/ROUGE could favor repetitive answers such as ChatGLM-2-6B despite its much lower proposed score of 10.11/100.

One-shot examples usually helped more consistently than long cheat sheets. In contrast, knowledge augmentation could become unstable when the provided reference material exceeded the model’s effective context window. These results suggest that “more medical text” is not automatically better evidence.

Benchmark comparison

We measured two capabilities: text-based medical reasoning through long-form OpenQA, and multimodal patient diagnosis through image-based MCQ. PatientQA is more demanding than common benchmarks because it evaluates diagnostic responses and asks MLLMs to combine patient context with medical images. The results show that models remain weak on both tasks: most multimodal models perform only marginally above random selection, while even strong models remain below the 60% examination threshold on patient-summary diagnosis.

Inspecting visual attention

We visualize attention to test whether MLLMs actually focus on medically meaningful content when processing medical images. We find that current models do not attend to medical content as reliably as they attend to natural-image content, revealing a gap in multimodal medical understanding.

Prompting results across the three tasks

One-shot examples generally improve medical history collection by helping models follow the reference format. Patient-summary diagnosis remains difficult: even strong models struggle to exceed the 60% examination threshold, and long cheat sheets can exceed the effective context length. For multimodal MCQ, chain-of-thought helps some models, but the gains are inconsistent and model scale alone does not guarantee better medical visual reasoning.

Metric comparison on OpenQA

Data collection

The dataset is built from printed medical exercises containing Medical License Examination (MLE) and OpenQA questions supplied by clinical doctors. Starting from more than 10,000 MCQs, questions are separated according to whether they mention an image. Image-dependent questions are then manually verified when they share images or require context from previous questions, yielding approximately 665 multimodal questions. The OpenQA subset contains 487 questions for medical history collection and patient-summary diagnosis, with reference answers and diagnostic cheat sheets for evaluation.

Conclusion

PatientQA complements existing medical benchmarks with two patient-centered evaluation settings. Its OpenQA subset uses step-by-step reasoning to evaluate long-generated diagnostic texts, while its multimodal MCQ subset tests whether MLLMs can use medical images for patient diagnosis. The experiments show that current models still struggle to reach expert-level performance, and that long knowledge-augmentation material can disorient models rather than reliably improve diagnosis. These findings highlight the complexity of medical evaluation and the need for stronger methods for handling patient context and multimodal information.

Links

• Technical report (PDF)
• Code / dataset: not publicly released