Toward trustworthy medical question answering: Multi-metric evidence for RAG-enhanced Large Language Models
Abstract
The integration of large language models (LLMs) into healthcare is hindered by their tendency to generate hallucinations, a critical issue in medical question answering (MedQA). Retrieval-Augmented Generation (RAG) frameworks mitigate this by grounding LLMs in verifiable knowledge. In this study, a weight-preserving RAG pipeline was implemented and evaluated to enhance factual fidelity without fine-tuning the base model. GPT-4 was combined with a dense passage retriever, employing Facebook AI Similarity Search (FAISS) with the BAAI/bge-small-en encoder, and compared against a non-retrieval GPT-4 baseline. Evaluation was conducted on a held-out set of 1,000 questions using widely adopted computational metrics, including BLEU-1/2/3/4, ROUGE-L, METEOR, text-level F1, and Exact Match (EM), complemented by qualitative case analysis. The RAG-enhanced configuration significantly outperformed the baseline across all metrics. Most notably, a 40.1% relative improvement in BLEU-4 (0.3015 → 0.4224) and a 14.43% gain in METEOR (0.3880 → 0.4440) were observed, while the Exact Match rate more than doubled from 0.02 to 0.05. Qualitative analysis further indicated fewer omissions and more faithful terminology use, consistent with the quantitative results. These findings provide robust empirical evidence that a RAG architecture fundamentally enhances the factual reliability of GPT-4 in medical QA. The proposed framework offers a practical and scalable strategy to reduce hallucinations without task-specific fine-tuning, underscoring the potential of retrieval-augmented models to support trustworthy AI-assisted healthcare applications.