Automatic medical report generation: a comprehensive review of methodologies and applications

Early paradigms: rule-driven approaches

Before deep learning became mainstream, early AMRG systems relied primarily on well-defined rules and pre-built resource libraries. While these approaches had limited flexibility, they laid the foundation for the emergence of more complex models. They were primarily categorized into two strategies: template-based and retrieval-based.

Retrieval-based methods

Retrieval-based generation methods operate on the assumption that similar medical images correspond to similar report descriptions. These methods construct a large-scale database of image-report pairs. When presented with a new image, the system retrieves the most similar image and its corresponding report based on predefined similarity rules and matching schemes. The final report is then generated by selecting, combining, or modifying relevant fragments from the retrieved reports (Fig. 5). Unlike template-based methods, retrieval-based approaches utilize a vast repository of annotated medical reports and case records, which primarily rely on image-text similarity matching to generate reports. Liu et al. (2019) demonstrated that retrieval-based methods achieved higher recall rates compared to other approaches, even when simply assigning the most similar retrieved report to a new image. Given their reduced reliance on large-scale training datasets and computational resources, retrieval-based methods have been widely explored, particularly in data-limited scenarios.

Zhang et al. (2018) proposed a multi-label classification scheme combined with transfer learning to address the annotation challenge in the ImageCLEF2018 caption task. Their retrieval-based approach generated captions by retrieving similar images using color and texture features, then aggregating the captions of top-ranked matches. Yang et al. (2021a) introduced MedWriter, a model based on a hierarchical retrieval mechanism designed to automatically extract report and sentence-level templates for report generation. MedWriter innovatively proposed three key modules: (1) Visual-Language Retrieval (VLR) to retrieve the most relevant report template for a given image, (2) Language-Language Retrieval (LLR) to select relevant sentences based on the previously generated descriptions to ensure logical coherence within the report, and (3) Hierarchical Language Decoder to fuse image features with the retrieved report and sentence features to generate clinically meaningful medical reports. Endo et al. (2021) proposed CXR-RePaiR, a retrieval-based radiology report generation method that utilized a pre-trained contrastive language-image model. They argued that report generation should be reframed as a retrieval task rather than a traditional image captioning or language generation task so as to leverage zero-shot learning and the limited space of possible findings and diagnoses inherent in medical reports. However, while their ablation studies analyzed how factors such as pre-training techniques, retrieval corpus type, and the number of selected sentences impacted model performance, their method did not address a key limitation of retrieval-based approaches: the inability to predict rare pathologies that were absent from the reference corpus. Notably, Jeong et al. (2024) adopted a similar image-text retrieval approach to that of Endo et al. (2021) for radiology report generation. However, unlike (Endo et al., 2021) who used two pre-trained unimodal encoders to compute similarity scores, Jeong et al. (2024) attempted to overcome the problem of retrieval-based attempts often retrieving reports unrelated to the input image by using a multimodal encoder that fuses image and text representations to achieve image-text matching, thereby improving retrieval accuracy. However, this approach suffers from significant data bias, and whether it can achieve significant results outside of the domain remains unknown. Syeda-Mahmood et al. (2020) introduced a domain-aware retrieval framework that leveraged fine-grained lesion descriptions and constructed a multi-model feature pyramid to enhance retrieval quality. Although their approach improved clinical precision, the system required extensive domain knowledge and manual feature engineering, which hinders scalability. Sun et al. (2024) proposed a fact-aware multimodal retrieval-augmented pipeline (FactMM-RAG) for radiology report generation. Their approach used RadGraph (Jain et al., 2021) to annotate chest X-ray reports and extract clinically relevant concept pairs. They then used a pre-trained multimodal encoding architecture to perform dense retrieval of paired radiology reports. While still relying on a report corpus, this approach significantly improved clinical effectiveness.

Although retrieval-based methods have demonstrated promising results in AMRG by leveraging large amounts of annotated data to efficiently match and generate medical reports, they remain fundamentally data-dependent. As Endo et al. (2021) highlighted, clinical scenarios were highly complex and dynamic. When encountering emerging diseases or intricate medical images, retrieval-based approaches often struggle to generate accurate and reliable reports. Similarly, Chen et al. (2020) pointed out that the pathological heterogeneity of real-world medical data posed significant challenges in constructing large-scale retrieval databases, which further limited the adaptability of retrieval-based models. Nevertheless, with the growing availability of medical data and advancements in retrieval algorithms, these methods are expected to become increasingly flexible and accurate and offer improved adaptability to diverse clinical settings.

Template-based and retrieval-based methods face inherent challenges in terms of flexibility and scalability. Their limited ability to handle the high variability and complexity of clinical scenarios has paved the way for more flexible, data-driven end-to-end generative approaches.

The deep learning era: the rise of end-to-end generation methods

End-to-end generation methods learn the mapping relationship between medical images and textual descriptions and demonstrate strong performance in processing complex visual content and generating fluent and natural language reports. With the rapid advancement of deep learning technologies, end-to-end generation has progressively supplanted rule-based approaches and become the predominant paradigm in AMRG. Consequently, this framework has represented the mainstream direction for most contemporary research in AMRG. This section provides a systematic analysis of the most employed architectures in end-to-end generation methods and highlights their key components, advantages, and limitations.

The encoder-decoder model is the most widely used architecture in AMRG. While various encoder-decoder configurations have been developed, they all follow the same fundamental principle. This architecture consists of two key components: the encoder and the decoder. The encoder processes the input data and maps it to a fixed-size hidden state vector. In the context of medical image report generation, the encoder is responsible for extracting meaningful features from medical images, and it typically utilizes CNN as the primary image encoder. On the contrary, the decoder is primarily used to generate descriptive text corresponding to the input image, and the decoder implementation commonly employs Recurrent Neural Network (RNN) and their variants, such as Long Short-Term Memory (LSTM) networks and Gated Recurrent Unit (GRU). The application of the encoder-decoder architecture in AMRG is illustrated in Fig. 6.

The application of the encoder-decoder model architecture to the AMRG task was originally inspired by the Show-And-Tell model proposed by Vinyals et al. (2015) in the field of natural image captioning. This model employed CNN for image encoding and LSTM for text generation, which successfully produced descriptive sentences for natural images with outstanding performance. Hence, the introduction of the Show-and-Tell model laid the foundation for adopting the encoder-decoder architecture in AMRG. Shin et al. (2016) were the first to apply the encoder-decoder framework to AMRG. They developed a CNN-RNN deep learning model capable of simultaneously detecting diseases and annotating contextual information, such as location, severity, and affected organs, from medical images. Their approach involved using image annotations to extract disease names, which were then used to train a CNN. Subsequently, an RNN was then trained to generate descriptive context based on the extracted deep CNN features. Singh et al. (2019) extended this approach by utilizing the Inception-v3 model (Szegedy et al., 2016) as the CNN encoder and a multi-level stacked LSTM as the decoder, which converted medical image features into radiology reports, and their findings demonstrated the potential of stacked RNN (SRNN) for generating radiology reports. Building on the concept of SRNN, researchers have also explored hierarchical RNN (HRNN) as decoders. The core idea of HRNN is to introduce multi-level hierarchical processing for sequential data, enabling them to capture language features more effectively and generate longer and more coherent texts (Krause et al., 2017). For example, Jing, Xie & Xing (2017) argued that single-layer LSTMs had limited modeling capacity for long word sequences. To address this, they leveraged the compositional nature of medical reports and introduced a hierarchical LSTM for long-text generation. In this framework, after CNN encoding was completed, the context vector was passed to a sentence-level LSTM, which expanded over several steps to produce topic vectors that represented the semantics of each sentence to be generated. These topic vectors were then fed into a word-level LSTM, which generated fine-grained word sequences to form sentences. Wang et al. (2022a) built on this approach by selecting DenseNet-121 (Huang et al., 2017) as the encoder network backbone and implementing a two-layer LSTM decoder similar to the one proposed by Jing et al. In addition, they also introduced a graph convolutional neural network (GCN) module, which integrated prior knowledge from text mining to enhance the medical accuracy of generated radiology reports. Subsequent studies by Huang et al. (2019), Yuan et al. (2019) and Harzig et al. (2019) further advanced hierarchical RNN in various ways.

It is evident that the majority of encoder-decoder-based research in AMRG has centered around LSTM networks that can achieve state-of-the-art (SOTA) results. This success is attributed to the gating mechanisms of LSTM—the input gate, forget gate, and output gate—which regulate information flow and effectively mitigate the gradient vanishing problem (the LSTM architecture is illustrated in Fig. 7). These properties make LSTM well-suited for tasks requiring the capturing of long-term dependencies. However, despite its ability to model long-range dependencies, LSTM-based text generation may still lead to word omissions in the generated report sentences (Najdenkoska et al., 2021). Compared to LSTM, the GRU offers a more streamlined architecture, improved computational efficiency, and a similar gating mechanism. Consequently, GRU is often a preferable choice in scenarios with limited data availability or constrained computational resources, as it can achieve comparable or even superior performance. Akbar et al. (2023) employed GRU as the decoder in their CNN-GRU model architecture. Their approach fed both the image vector and text embedding layers into the GRU decoder during training, demonstrating promising results in generating short and grammatically correct sentences with fluent medical terminology.

These developments illustrate the evolution of encoder-decoder models in AMRG: from simple CNN-RNN pipelines to more sophisticated hierarchical architectures that attempt to balance fluency and clinical correctness. Despite the enormous success of encoder-decoder models, their core bottleneck lies in the fact that the encoder must compress all input information into a fixed-length context vector. For information-rich medical images and lengthy reports, this compression process inevitably leads to information loss, especially when dealing with long-range dependencies. This fundamental limitation motivates the introduction of the attention mechanism.

The transformer revolution: redefining sequence modeling

In order to solve the information bottleneck problem of the traditional encoder-decoder architecture, researchers have explored various attention mechanisms (Niu, Zhong & Yu, 2021), enabling the model to selectively “look back” at different regions of the image during generation. This paradigm ultimately culminated in the Transformer, which replaced the sequential RNN structure entirely with a more powerful self-attention mechanism, proving far more effective at managing these complex relationships.

Attention-based models

The attention mechanism allows the decoder to dynamically focus on different regions of the input image when generating each word (Fig. 8), rather than relying on a single fixed vector. This enables the model to more effectively capture key visual areas and subtle pathological changes. However, different variants approach this goal from distinct perspectives, including refinement of spatial focus, hierarchical alignment with medical semantics, or high-order feature interactions. Representative advances and their respective contributions are summarized below.

Zhang et al. (2017) were among the first to incorporate attention mechanisms into medical image models. They proposed a diagnostic captioning model for bladder cancer images, named MDNET, which integrated both image and language models. Their approach introduced an auxiliary attention sharpening (AAS) module that was designed to refine the original attention mechanism and ensured a stronger focus on informative regions. This model achieved SOTA performance on a dataset consisting of pathological bladder images and their corresponding diagnostic reports. While effective, its reliance on handcrafted attention sharpening limited scalability to more complex imaging modalities. Subsequent studies extended attention to capture richer local or multimodal contexts. Han et al. (2021) introduced a neural-symbolic learning (NSL) framework for spinal radiology report generation, integrating deep neural learning with symbolic reasoning. Their local semantic attention mechanism modeled visual features as symbolic nodes, enabling probabilistic weighting of multiple image regions. This design improved multi-perspective information capture, though its symbolic abstraction may struggle with highly heterogeneous imaging data. Jing, Xie & Xing (2017) advanced this line of work with a co-attention mechanism in a multi-task framework, jointly attending to visual sub-regions and semantic embeddings. This approach facilitated precise localization of abnormalities but increased computational cost. Park et al. (2020) further extended co-attention with collaborative attention in mDiTag, which combined feature differences and label information with a hierarchical LSTM decoder to improve the accuracy of anomaly detection. You et al. (2021) proposed aligned hierarchical attention (AHA), which hierarchically aligned visual regions with disease labels. By leveraging the structured nature of radiology reports, AHA captured disease-relevant features at multiple granularities, although its hierarchical design requires careful tuning to avoid overfitting to label distributions. Wang et al. (2022c) introduced memory-enhanced sparse attention (MSA) to enhance the ability of the model to capture fine-grained visual differences. Specifically, they employed bilinear pooling to model high-order interactions between fine-grained image features, while simultaneously generating sparse attention to better adapt to radiological images with fine-grained details. Xu et al. (2023) proposed M-linear attention, which was also based on bilinear pooling blocks, to enhance intra- and inter-modality reasoning. While both methods improved fine-grained feature modeling, their reliance on bilinear pooling highlights the trade-off between representational richness and efficiency. Song et al. (2022) introduced a cross-modal contrast attention (CMCA) model designed to capture visual and semantic information from similar medical cases. CMCA employed visual contrast attention to identify unique abnormal regions and cross-modal attention to dynamically align textual semantics with image features during report generation. Although the addition of CMCA can enhance contextual relevance, it remains sensitive to the quality and diversity of retrieved clinical cases, raising questions about robustness in underrepresented conditions.

Collectively, these studies demonstrate how attention mechanisms evolve from fine-grained local focus to achieving multi-level semantic alignment and modeling complex high-order interactions. While attention mechanisms significantly improve the accuracy and interpretability of AMRG systems, different variants face trade-offs between computational complexity, reliance on high-quality annotations, and generalization to diverse clinical conditions. This comparison suggests that no single attention mechanism universally addresses the challenges of AMRG, but each contributes to bridging the vision-language gap.

The dominance of the transformer architecture

The attention mechanism represents a significant advancement in sequence-to-sequence models, enabling them to dynamically focus on the most relevant parts of the input when generating each output element. Building on this concept, the introduction of the Transformer (Vaswani et al., 2017) architecture has marked a major breakthrough, as it fully leverages the self-attention mechanism to redefine the technical paradigm of medical image report generation. By effectively capturing long-range dependencies, this facilitates both image features and textual information processing, enhances contextual modeling, and ultimately improves report generation quality. As a result, the Transformer architecture has progressively supplanted various deep learning frameworks and led to a transformative shift in the advancement of AMRG.

The Transformer architecture in AMRG is typically combined with a CNN encoder (Fig. 9). Chen et al. (2020) proposed a memory-driven Transformer model, R2Gen, for radiology report generation, incorporating relational memory to retain information from previous text generation steps. This innovation helped alleviate the problem of context fragmentation in long medical reports. However, the reliance on CNN-based feature extraction limited its ability to capture long-range visual dependencies. R2GenCMN, another model by Chen et al. (2022), extended R2Gen by integrating Cross-Modal Memory Networks (CMN) to improve cross-modal interaction and alignment. CMN stored cross-modal information in a memory matrix, which retrieved the most relevant memory vectors based on input visual and textual features. These weighted memory vectors were then used to generate a response, which was subsequently fed into the Transformer encoder and decoder layers to produce the final report. While CMN improved alignment between visual and textual features, it introduced higher computational costs and increased model complexity, which may hinder clinical deployment. Zhang et al. (2022) combined a Transformer text encoder with CNN-derived image features and further integrated a medical knowledge graph to enrich disease representation. This hybridization improved semantic accuracy, but its performance was constrained by the incompleteness and domain-dependence of medical knowledge graphs. Huang, Zhang & Zhang (2023) introduced the Knowledge-injected U-Transformer (KiUT), designed to learn multi-level visual representations while dynamically integrating contextual and clinical knowledge to improve word prediction. Their model incorporated U-connections between encoder and decoder layers, which enabled feature aggregation at all decoder layers. Furthermore, by injecting visual, contextual, and clinical knowledge signals, their approach generated clinically relevant reports that better align with real-world medical scenarios. Several other studies have also explored the application of Transformers in AMRG (Yang et al., 2022b; You et al., 2022; Zhang et al., 2023a; Qin & Song, 2022) and Hou et al. (2023c). However, it is important to note that while these models incorporated Transformers, they did not completely replace CNN. One study (Wang et al., 2022b), in contrast, abandoned CNN entirely and developed a pure Transformer-based model that eliminated the need for CNN. Instead of relying on CNN for image feature extraction, they employed a Transformer-based visual encoder to overcome the limited receptive field of CNN and enable the model to learn long-range visual dependencies more effectively. Ye et al. (2024) proposed a multimodal dynamic traceability learning framework, DTrace, designed to supervise semantic validity and optimize the dynamic learning strategy for generated reports. As a dual-stream Transformer model, DTrace maintained separate feature representations for images and text while allowing effective cross-modal information exchange through attention mechanisms. This approach ensures both modality independence and robust feature interaction, but the model‘s reliance on dynamic traceability learning introduces training instability and requires careful hyperparameter tuning. In contrast to R2Gen, which focused on retaining temporal context using a memory network to ensure report coherence, DTrace focuses on cross-modal semantic validity by dynamically tracing the generated text back to specific image features. This shift in focus yields concrete performance gains, as DTrace shows improvements over R2Gen on MIMIC-CXR in both linguistic metrics (e.g., BLEU-1: 0.392 compared to 0.353) and, more importantly, in clinical efficacy (CE F1: 0.391 compared to 0.276).

The success of Transformer lies in its powerful context modeling capabilities and parallel processing efficiency, making it particularly suitable for generating complex and logical radiology reports. However, training large Transformer models requires massive computing resources and data, which has driven research towards the use of pre-trained large language models.

The new frontier: adaptation of large language model

With the emergence of large language model (LLM) with billions of parameters, their superior semantic understanding and text generation capabilities have brought new transformative opportunities to AMRG. Applying LLM to AMRG is an emerging and innovative research direction. LLM follow a pre-training and fine-tuning strategy, allowing for effective adaptation to specific domains through post-training techniques such as domain adaptation. While pre-training does not directly optimize performance for a specific task, subsequent fine-tuning enables LLM to specialize in AMRG. Furthermore, their zero-shot and few-shot learning capabilities significantly reduce the dependency on large-scale annotated datasets, thereby mitigating data scarcity challenges in medical image report generation.

Despite the high expectations for LLM in AMRG, their extensive parameter count presents a significant challenge in terms of computational resources. A common approach to address this issue is to freeze the LLM and employ a lightweight network to map image features into the text feature space of the LLM. This strategy reduces computational overhead and enhances model adaptation efficiency. Zhang et al. (2024c) proposed MSMedCap, a medical image captioning model based on a dual image encoder and hybrid semantic learning, designed to enhance feature encoding and capture both global and fine-grained details in medical images. Their approach utilized two visual Transformers pre-trained with Contrastive Language-Image Pre-training (CLIP) and the Segment Anything Model (SAM) as image encoders. The extracted image features were then aligned and aggregated through Dual Query Transformers (Q-Former) and linear projection layers before being fed into the frozen LLM to generate medical captions via text prompts. Q-Former, originally proposed by Li et al. (2023a), served as a trainable mapping module that facilitated the integration of frozen image encoders with frozen LLM. However, the effectiveness of freezing both the visual encoder and LLM in AMRG tasks remained an open research question, as it necessitated large-scale aligned medical data for pre-training. Lu et al. (2023) challenged the frozen LLM paradigm, arguing that fine-tuning the visual model could enhance AMRG performance. They introduced a two-stage fine-tuning strategy, in which the visual encoder, mapping network, and text decoder were fine-tuned in stages. Initially, the visual encoder remained frozen for one epoch, allowing the mapping network to align information between the two modalities. Subsequently, the visual encoder was unfrozen for the remaining epochs. This progressive fine-tuning approach ensured that visual features remained consistent with the text embedding space of LLM, thereby improving clinical accuracy across multiple scales. Liu et al. (2024b) proposed a method to guide LLM to generate radiology reports through in-domain instance induction and a coarse-to-fine decoding process. They employed MiniGPT-4 (Zhu et al., 2023a) as the visual encoder to extract latent visual representations. The text generator of MiniGPT-4 was then guided by instance retrieval and contrastive semantic ranking, aligning the LLM with radiology-specific text reports to generate coarse initial reports. A subsequent coarse-to-fine decoding process further refined these reports to produce clinically coherent and structured outputs, and the fine-tuned LLM demonstrated promising potential in AMRG. Wang et al. (2024c) chose to use MAMBA (Gu & Dao, 2023), a time-varying state-space model based on a selection mechanism, as a visual backbone. Unlike Transformers, MAMBA offers linear computational complexity, significantly reducing computational costs while achieving comparable performance to Transformer-based vision models. In addition, before extracting visual tokens, their model performed context retrieval on mini-batch samples during training that improved feature representation and discriminative learning. Ultimately, the LLM played a central role in report generation, integrating visual and textual residuals from positive and negative context samples along with the corresponding text tokens to generate clinically relevant reports.

LLM demonstrates great potential for generating highly coherent, semantically rich reports. However, LLM also presents unprecedented risks and challenges, representing chasms that must be overcome before their practical clinical application. First, clinical “hallucinations” are the most critical risk of LLM in the medical field. Unlike simple model errors, “hallucinations” occur when the model generates fluent, seemingly plausible descriptions that are completely inconsistent with the imaging facts. The fundamental reason for this is that LLM is essentially probabilistic text generators, not factual reasoning engines. They are trained to predict the most likely next word, not to verify the truth of a statement. Therefore, due to the frequent presence of irrelevant information in the training set, the model may “fabricate” a common pathology that is not present in the image or miss a rare but critical finding. In clinical practice, a “hallucinated” false-positive result can lead to unnecessary patient anxiety and expensive testing, while a false-negative (omission) can have catastrophic consequences. Second, directly applying a model trained on a general corpus to radiology presents significant challenges. Radiology reports typically contain a large number of precise anatomical and pathological terms and follow a specific report structure. A general LLM without sufficient domain adaptation may replace precise medical terminology with ambiguous everyday terms or fail to understand the diagnostic logic implicit in radiology reports, resulting in the generation of clinically unreliable or even misleading content. Therefore, simple text fine-tuning is insufficient. Future fine-tuning strategies could incorporate structured medical knowledge, forcing the model to learn the logical chain and causal relationships of diagnosis, rather than simply mimicking the report’s writing style. Furthermore, leveraging retrieval-augmented generation (RAG) technology to “anchor” its output on a real clinical knowledge base may be one of the most promising approaches to addressing the problem of missing factual evidence.

Natural language generation (NLG) metrics

NLG metrics were originally designed for natural language processing tasks. However, due to the lack of dedicated evaluation metrics for AMRG, NLG metrics have been widely adopted in this domain. These evaluation methods measure the similarity between the generated reports and ground-truth reports based on word overlap. The most commonly used NLG metrics include:

1. Bilingual Evaluation Understudy (BLEU) (Papineni et al., 2002): BLEU is a metric used to evaluate the closeness between a candidate (machine) translation and a reference (true) translation and is widely used in machine translation and text generation tasks. In the AMRG task, BLEU measures the similarity between the generated report and the true report by analyzing the overlap of n-grams of length up to 4-word sequences. BLEU scores range from 0 to 1, with values approaching 1 indicating greater consistency between the generated and reference reports. BLEU-1 measures the overlap of unigrams, while BLEU-2, BLEU-3, and BLEU-4 are used for bigrams, trigrams, and tetragrams, respectively. It simultaneously considers different parameters, such as text length, vocabulary selection, and word order, and the similarity between the real report to penalize or reward the generated text. BLEU can be expressed in Eq. (1):

(1) $$\mathrm{B}\mathrm{L}\mathrm{E}\mathrm{U}=BP\cdot \exp \left(\sum _{\mathrm{n}=1}^N{w}_n\log \left[\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}(\mathrm{n})\right]\right).$$

BP is referred to as the brevity penalty, which enables the selection of candidate translations that are most likely to be close to the true translation in terms of length, vocabulary choice, and word order. Precision(n) represents the precision of different n-grams, and w_n is the weight of the n-gram that is conventionally set to 1/N with N = 4. The calculation of BP is shown in Eq. (2):

(2) $$BP=\{\begin{array}{cc}1\hfill & \mathrm{i}\mathrm{f}\mathrm{c}>\mathrm{r}\hfill \\ e^{\left(1-\frac{r}{c}\right)}\hfill & \mathrm{i}\mathrm{f}\mathrm{c}\le \mathrm{r}\hfill \end{array}.$$where c is the length of the model-generated text, and r is the length of the reference text.

2. Recall-Oriented Understudy for Gisting Evaluation (ROUGE) (Lin, 2004): ROUGE is a standardized tool for evaluating the quality of automatic text generation. It mainly measures its quality by comparing the overlap between the generated text and the reference text, with a particular focus on the recall of the generated text, that is, how much common part the generated text has with the reference text.

ROUGE-L is the most commonly used variant, which is calculated based on the longest common subsequence (LCS) and can better reflect the grammatical structure and sentence fluency of the report. The formula is displayed in Eq. (3):

(3) $$ROUGE-L={\displaystyle \frac{(1+{\beta }^2)\cdot R_{lcs}\cdot P_{lcs}}{R_{lcs}+{\beta }^2\cdot P_{lcs}}}$$where β is a hyperparameter, which can be modified according to the needs of the specific task. R_lcs and P_lcs represent the recall rate based on LCS and the accuracy rate based on LCS, respectively.

In the AMRG task, the ROUGE metric is employed to evaluate whether the generated report is consistent with the reference report annotated by experts, especially focusing on the fluency and relevance between sentences. The ROUGE score ranges from 0 to 1, and the higher the score, the better the relevance between the generated text and the reference text.

3. Consensus-based Image Description Evaluation (CIDEr) (Vedantam, Lawrence Zitnick & Parikh, 2015): CIDEr is an automatic evaluation metric mainly used for image description generation tasks. It uses the Term Frequency Inverse Document Frequency (TF-IDF) weighted n-gram measure to generate captions and reference letters by effectively accounting for syntax, saliency, and accuracy. The cosine similarity between them is used to evaluate the quality of the generated image description. TF assigns higher weights to n-grams that frequently appear in reference sentences describing images, while IDF assigns lower weights to n-grams that frequently appear in all images in the dataset. The CIDEr score of an n-gram of length n can be expressed using Eq. (4):

(4) $$\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}{\mathrm{r}}_n(c_i,S_i)={\displaystyle \frac{1}{m}\sum _j{\displaystyle \frac{g^n(c_i)\cdot g^n(s_{ij})}{\Vert g^n(c_i)\Vert \Vert g^n(s_{ij})\Vert }}}$$where gⁿ(s_ij) is the vector formed by g_k(s_ij) corresponding to all n-grams of length n, and ||gⁿ(s_ij)|| is the size of the vector gn(sij). Similarly for gⁿ(c_i). The TF-IDF weight g_k(s_ij) of each n-gram ω_k (ngram ω_k is a set of one or more ordered words) can be calculated using Eq. (5):

(5) $$g_k\left(sij\right)={\displaystyle \frac{h_k(s_{ij})}{{\sum }_{{\omega }_l\in \mathrm{\Omega }}{h}_l(s_{ij})}\log \left({\displaystyle \frac{|I|}{{\sum }_{I_p\in I}min(1,{\sum }_q{h}_k(s_{pq}))}}\right),}$$

For a candidate sentence c_i, the number of times the n-gram ω_k appears in the reference sentence s_ij is denoted as h_k(s_ij). Where Ω is the vocabulary size of all n-grams, and I is the set of all images in the dataset.

Finally, CIDEr is calculated by combining the scores of n-grams of different lengths (usually uniform weight wn = 1/N works best) as shown in Eq. (6):

(6) $$\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}\mathrm{r}(c_i,S_i)=\sum _{n=1}^N{w}_n\mathrm{C}\mathrm{I}\mathrm{D}\mathrm{E}{\mathrm{r}}_n(c_i,S_i).$$

CIDEr is particularly important in the field of image description generation because it not only examines the degree of vocabulary matching but also emphasizes the diversity, accuracy, and comprehensive capture of image content.

4. Metric for Evaluation of Translation with Explicit Ordering (METEOR) (Banerjee & Lavie, 2005): METEOR aims to overcome the limitations of traditional evaluation metrics such as BLEU. In particular, it better reflects the semantic consistency of text through flexible treatment of synonyms, word form changes, and word order, rather than relying solely on literal vocabulary overlap. This metric is developed with the explicit aim of surpassing the evaluation quality of BLEU, especially in dealing with semantic matching and synonym replacement. Its core methodology incorporates flexible matching rules that consider the influence of synonyms, word form changes, and word order in the text, which ensures that the evaluation can reflect the true quality of the generated text. This method calculates the harmonic mean of unigram precision and recall, which can be mathematically expressed via Eq. (7):

(7) $$METEOR=F_{mean}\cdot (1-Penalty)$$where F_mean is the harmonic mean of precision and recall. Penalty is used to penalize candidate sequences with poor word order. The penalty function examines the order and relative position of the vocabulary matching between the candidate text and the reference text, and it applies a score reduction if the matching vocabulary order does not match the reference text. The calculation formula for Penalty is displayed in Eq. (8):

(8) $$Penalty=0.5\cdot {\left({\displaystyle \frac{\mathrm{\#}\mathrm{c}\mathrm{h}\mathrm{u}\mathrm{n}\mathrm{k}\mathrm{s}}{m}}\right)}^3$$where #chunks represent the number of phrases in the candidate translation and the reference translation that are relatively consistent in position and continuously match, and m represents the total number of matches between the candidate words and the reference words.

In the AMRG task, METEOR effectively assesses similarity between the generated report and the expert annotation report by evaluating synonym matching, word forms, word order, grammatical consistency, and the textual diversity, leading to its widespread adoption.

5. Bidirectional Encoder Representations from Transformers (BERTScore) (Zhang et al., 2019): BERTScore is a language generation evaluation metric based on pre-trained BERT contextual embeddings. Different from traditional n-gram overlap-based evaluation metrics (such as BLEU, ROUGE, METEOR, etc.), BERTScore considers not only the matching of words but also the context of words to capture deeper semantic information. In the original article (Zhang et al., 2019), the authors demonstrated that BERTScore was more correlated with human judgment and provided stronger model selection performance than existing indicators. The full process of its calculation is as follows:

First, BERTScore processes both the reference and the candidate texts through a pre-trained BERT model to obtain the BERT representation of each word in a context-sensitive word vector. Then cosine similarity is used to calculate the similarity of the two-word vectors via Eq. (9):

(9) $$\mathrm{C}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}(x,y)={\displaystyle \frac{x\cdot y}{\Vert x\Vert \Vert y\Vert }.}$$

Among them, x and y represent the BERT representation vector of a word in the candidate text and the reference text, respectively, and ||x|| and ||y|| represent the Euclidean norm of the two vectors. Usually, a pre-normalized vector is used, and the calculation can be simplified to the inner product x • y. BERTScore employs cosine similarity to calculate the following three metrics: Precision (normalized maximum similarities from candidate to reference tokens), Recall (normalized maximum similarities from reference to candidate tokens), and their harmonic mean as F1-score. The final BERTScore usually uses the F1 value as a comprehensive score, which indicates the semantic alignment between the candidate text and the reference text. The calculation of the three metrics is shown in Eqs. (10)–(12):

(10) $${\mathrm{P}}_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}={\displaystyle \frac{1}{|C_{\mathrm{g}\mathrm{e}\mathrm{n}}|}\sum _{c_{\mathrm{g}\mathrm{e}\mathrm{n}}\in C_{\mathrm{g}\mathrm{e}\mathrm{n}}}\underset{c_{\mathrm{r}\mathrm{e}\mathrm{f}}\in C_{\mathrm{r}\mathrm{e}\mathrm{f}}}{max}\mathrm{C}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}(c_{\mathrm{g}\mathrm{e}\mathrm{n}},c_{\mathrm{r}\mathrm{e}\mathrm{f}})}$$

(11) $${\mathrm{R}}_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}={\displaystyle \frac{1}{|C_{\mathrm{r}\mathrm{e}\mathrm{f}}|}\sum _{c_{\mathrm{r}\mathrm{e}\mathrm{f}}\in C_{\mathrm{r}\mathrm{e}\mathrm{f}}}\underset{c_{\mathrm{g}\mathrm{e}\mathrm{n}}\in C_{\mathrm{g}\mathrm{e}\mathrm{n}}}{max}\mathrm{C}\mathrm{o}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{S}\mathrm{i}\mathrm{m}\mathrm{i}\mathrm{l}\mathrm{a}\mathrm{r}\mathrm{i}\mathrm{t}\mathrm{y}(c_{\mathrm{g}\mathrm{e}\mathrm{n}},c_{\mathrm{r}\mathrm{e}\mathrm{f}})}$$

(12) $$F1=F_{BERT}=2\times {\displaystyle \frac{{\mathrm{P}}_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}\times {\mathrm{R}}_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}}{{\mathrm{P}}_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}+{\mathrm{R}}_{\mathrm{B}\mathrm{E}\mathrm{R}\mathrm{T}}}}$$where C_gen and C_ref are the word sets in the candidate text and the reference text, respectively.

Due to its foundation on the BERT pre-trained model, BERTScore exhibits adaptability across languages, fields, and tasks. In particular, BERTScore performs well in certain AMRG tasks that require deep semantic understanding. Although the BERTScore metric has been used in studies such as Syeda-Mahmood et al. (2020), Sun et al. (2024), Kapadnis et al. (2024), Miura et al. (2020), Cabello-Collado et al. (2024) and Leonardi, Portinale & Santomauro (2023), it is not included in the subsequent application evaluation due to its lack of widespread use.

6. Semantic Propositional Image Caption Evaluation (SPICE) (Anderson et al., 2016): SPICE is an automatic evaluation metric for natural image captions. SPICE analyzes the semantic structure of image descriptions and pays special attention to the semantic relationships conveyed in the descriptions. Rather than focusing merely on vocabulary overlap, it aims to assess if the generated description can accurately and completely express the semantic information in the image. Compared with traditional indicators such as BLEU, ROUGE, and METEOR, this metric provides higher precision in quality assessment of the generated description, suggesting its prospective widespread application in AMRG evaluation. It is worth noting that it is not tailored to medical terminology or domain-specific ontologies, so its ability to capture subtle diagnostic correctness remains limited.

Although NLG metrics are widely used due to their automation and convenience, they suffer from fundamental flaws when evaluating medical reports. These metrics focus on measuring lexical overlap rather than clinical semantic equivalence. This flaw leads to significant evaluation bias. For example, two clinically identical statements, “heart size and outline are normal” and “no cardiac enlargement was observed,” will have BLEU or ROUGE scores approaching zero due to minimal lexical overlap. Conversely, a model might achieve a high NLG score by repeating common “normal” descriptive terms in a dataset while completely missing a critical, potentially life-threatening abnormal finding in the imaging. This disconnect between “linguistic similarity” and “clinical accuracy” highlights the dangers of relying solely on NLG metrics. A report that scores highly on an NLG metric may be clinically useless or even harmful.

More recently, metrics such as RadGraph F1 and RadCliQ have been proposed to better reflect clinical accuracy by evaluating medical entities and relations within reports. These directions highlight the necessity of moving beyond surface-level lexical comparisons toward clinically grounded evaluation. Ultimately, no single metric fully captures linguistic quality and medical correctness. Therefore, the AMRG field urgently needs methods that can directly assess the correctness of diagnostic content, which has led to the development of clinical efficacy (CE) indicators.

Clinical efficacy (CE)

To fill the gap in NLG metrics for clinical accuracy assessment, researchers have developed CE metrics. Currently, the most popular CE assessment method is based on CheXpert, Irvin et al. (2019) a rule-based label extractor designed specifically for chest X-ray reports. Chen et al. (2020) applied CheXpert to generated reports, comparing their content against 14 predefined categories related to thoracic diseases and support devices. They then calculated tag-based precision, recall, and F1-scores and utilized them as CE metrics to evaluate model performance.

Currently, CheXpert-based evaluation methods have been widely adopted in AMRG research (Boag et al., 2020; Chen et al., 2022; Dalla Serra et al., 2022; Gao et al., 2024; Hirsch, Dawidowicz & Tal, 2024; Hou et al., 2023a, 2023b; Huang, Zhang & Zhang, 2023; Jin et al., 2024; Li et al., 2023b; Liu et al., 2024a, 2024b, 2021b, 2021c, 2019, 2023; Lovelace & Mortazavi, 2020; Lu et al., 2023; Moon et al., 2022; Najdenkoska et al., 2021, 2022; Nguyen et al., 2021; Nicolson, Dowling & Koopman, 2023; Nooralahzadeh et al., 2021; Qin & Song, 2022; Serra et al., 2023; Shang et al., 2022; Song et al., 2022; Tang et al., 2024; Tu et al., 2024; Wang et al., 2024a, 2023e, 2023f; Wu et al., 2022; Wu, Huang & Huang, 2023; Yan et al., 2021; Yan & Pei, 2022; Yang et al., 2023, 2022a, 2022b; Ye et al., 2024; Yi et al., 2024; Zhang et al., 2022, 2023b; Zhu et al., 2023b; Liu et al., 2025; Lang, Liu & Zhang, 2025; Wang et al., 2025; Li et al., 2025b; Tanno et al., 2025). Although CE metrics represent an important step toward bridging this gap by capturing disease-related findings, they are strongly tied to specific datasets (e.g., MIMIC-CXR) and are limited to predefined labels, making them less generalizable across diverse clinical settings.

Human evaluation

Human language is inherently complex and diverse, making the accurate and effective evaluation of AMRG models a challenging task. As a result, human evaluation is introduced as the most reliable assessment method to address the limitations of automatic evaluation metrics and ensure a more comprehensive assessment of report quality. In AMRG, this involves a subjective assessment of generated reports, which is conducted by medical experts such as radiologists and clinicians. Drawing from their professional expertise, these experts score and provide feedback based on key criteria, including accuracy, completeness, readability, and clinical value. Unlike automated metrics, human evaluation can identify subtle errors and potential issues that may be overlooked by computational methods. Given its ability to capture fine-grained details and clinical nuances, human evaluation plays a critical role in the development and optimization of AMRG systems, ensuring their practical applicability in real-world clinical scenarios.

We reviewed 18 related studies (Dalla Serra et al., 2022; Liu et al., 2021b, 2021c; Qin & Song, 2022; Li et al., 2019, 2023c, 2018; Liu et al., 2022, 2021d; Wang et al., 2020, 2024d; Yang et al., 2020a, 2021a; You et al., 2021; Zhang et al., 2024a, 2024b; Cao et al., 2023; Tanno et al., 2025) that incorporated human expert evaluation to assess the quality of AMRG models. Among them, Li et al. (2019, 2018) selected a subset of test samples from each method and conducted a survey via Amazon Mechanical Turk (MTurk). Participants were asked to choose the report that best matched the reference report from multiple generated outputs, based on key criteria such as language fluency, content selection, and accuracy of abnormal medical findings. Other studies (Liu et al., 2021b, 2021c; Qin & Song, 2022; Liu et al., 2022, 2021d; Wang et al., 2020; You et al., 2021; Zhang et al., 2024a), invited domain experts with relevant medical experience to evaluate the quality of generated reports. Specifically, evaluators were required to identify the most reasonable report from outputs generated by the research model, baseline models, or ground-truth reports, based on factors such as fluency of the generated text, comprehensiveness in capturing real anomalies, and factual fidelity of the report content. To ensure unbiased evaluation, the evaluation employed a blinded methodology wherein experts assessed reports without knowledge of their generative sources. Additionally, Wang et al. (2024d) and Yang et al. (2021a) invited experienced radiologists to evaluate generated reports and required them to assign ratings on a scale of 1 to 5, where higher scores indicated greater report acceptability. Notably, one study (Wang et al., 2024d) specified that reports should be evaluated based on three quality dimensions: accuracy, information content, and readability. Finally, in Dalla Serra et al. (2022) and Zhang et al. (2024b), evaluators assessed model performance by counting the errors and omissions in generated reports, which provided a quantitative measure of model reliability and accuracy.

While existing research confirms human evaluation as accurate, reliable, and effective, which commonly results in higher human preference ratings compared to baseline models, this methodology presents notable limitations. Most importantly, human evaluation is both time-consuming and resource-intensive (Yu et al., 2023). Moreover, clinicians and radiologists often operate under significant time constraints, making their participation in manual evaluations challenging. Requiring them to engage in extensive assessments not only disrupts their routine clinical work but also contradicts the primary goal of AMRG to alleviate the workload of medical professionals. Given these challenges, it is understandable that the majority of studies in our survey did not incorporate human evaluation, opting instead for automated evaluation metrics as a more scalable and practical alternative.

Finally, we added a comparison Table 1 to more intuitively summarize and contrast the characteristics, advantages, and limitations of various evaluation methods.

Benchmark datasets and applications

Benchmark datasets serve not only as the data foundation for research in AMRG but also as a standardized benchmark for comparing different models. These datasets are crucial resources for evaluating model performance and have significantly contributed to the advancement of medical image report generation. This section introduces some of the most commonly used benchmark datasets in AMRG. Table 2 provides an overview of the key characteristics and relevant information of these datasets.

COV-CTR

The COV-CTR (COVID-19 CT Report) dataset is a public dataset specifically designed for COVID-19 lung CT image analysis and Chinese radiology report generation. It consists of two primary components: 728 CT images, collected by Yang et al. (2020b) during the COVID-19 pandemic (349 COVID-19 cases and 379 non-COVID cases), and Chinese diagnostic reports, constructed by Li et al. (2023c), which include Findings and Impressions sections. To enhance clinical interpretability, radiologists have annotated abnormal terms and highlighted key areas within the reports. However, it is important to note that the dataset only contains binary labels (COVID-19 and Non-COVID) and does not include annotations for other pulmonary diseases.

We have identified nine relevant studies utilizing COV-CTR, with Table 9 summarizing their key findings. Among them, Wang et al. (2021) explicitly quantified the inherent visual-text uncertainty in multimodal radiology report generation at both the report level and sentence level. To address this challenge, they proposed a Sentence Matched Adjusted Semantic Similarity (SMAS) method, which provided a more precise measurement of semantic similarity between radiology reports. This approach achieved SOTA performance on COV-CTR. However, while their model effectively captured visual-text uncertainty at the report level, it did not explicitly map sentence-level uncertainty to specific uncertain regions in the image. Addressing this limitation could further enhance model performance. Moreover, Clinical-BERT, proposed by Yan & Pei (2022), also demonstrated strong results and achieved the second-highest scores across multiple evaluation metrics. Their approach represented the first attempt to incorporate domain-specific knowledge into pre-training for medical applications. Clinical-BERT introduced three domain-specific pre-training tasks: Clinical Diagnosis (CD), which performed multi-label classification for radiological findings; Masked MeSH Modeling (MMM), which predicted grid-based masked medical terms; and Image-MeSH Matching (IMM), which aligned image-text reports through sparse attention mechanisms. These pre-training objectives enabled better understanding of X-rays, significantly improving performance on downstream radiology tasks.

Table 9:

COV-CTR dataset: model performance comparison.

Model/Method	Author	Year	BLEU				ROUGE	CIDEr	METEOR	Human evaluation
			−1	−2	−3	−4
Wang et al. (2021)	Wang et al.	2021	0.810	0.766	0.721	0.679	0.790	2.371	–	–
Clinical-BERT (Yan & Pei, 2022)	Yan et al.	2022	0.759	0.713	0.675	0.641	0.737	1.218	–	–
ASGK (Li et al., 2023c)	Li et al.	2022	0.712	0.659	0.611	0.570	0.746	0.684	–	√
ICT (Zhang et al., 2023a)	Zhang et al.	2023	0.768	0.692	0.629	0.577	0.716	–	0.423	–
SA3RT (Song et al., 2024)	Song et al.	2024	0.703	0.644	0.601	0.566	0.691	–	0.419	–
HDGAN (Zhang et al., 2024a)	Zhang et al.	2024	0.765	0.705	0.656	0.617	0.758	–	0.442	√
MDAKF (Tan et al., 2024)	Tan et al.	2024	0.726	0.651	0.583	0.539	0.683	1.354	0.401	–
Wang et al. (2024d)	Wang et al.	2024	0.753	0.680	0.620	0.569	0.730	–	0.437	√
MicarVLMoE (Izhar et al., 2025)	Izhar et al.	2025	0.744	0.662	0.599	0.545	0.692	–	0.690	–

DOI: 10.7717/peerj-cs.3474/table-9

Note:

All values are taken from the original article. The best result is in bold, the second-best result is underlined. A dash (–) indicates a metric that the model did not evaluate. The table is grouped by year.

To further uncover research trends and patterns in the AMRG field, we analyzed the frequency of application of different methodologies across several major datasets, as shown in Table 10. A quantitative visualization of this distribution (Fig. 10), presented in the form of a heat map or stacked bar chart, revealing the applicability and popularity of different technical paradigms across different data sizes and types. The results indicate that the smaller but classic IU-Xray dataset has attracted the most diverse methodological research, ranging from early CNN-RNN models to the latest LLM, making it an ideal platform for testing new ideas. In contrast, the large-scale MIMIC-CXR dataset is a clear battleground for Transformer-based models, thanks to their requirement for massive amounts of data for effective training. Transformer-based methods dominate across all datasets, especially on large datasets like MIMIC-CXR, confirming their status as the current SOTA paradigm. Although LLM-based methods are still relatively few in number, they have been applied to all major datasets, demonstrating that this is a rapidly emerging research direction with great potential. Rule-driven methods have the fewest applications, primarily concentrated in early research or as part of hybrid models, reflecting their decline from mainstream practice in the era of end-to-end deep learning.

Table 10:

Frequency of application of different methods on several major datasets.

Dataset	CNN-RNN	Transformer	LLM	Rule-driven	Total
IU-Xray	27	55	12	7	101
MIMIC-CXR	11	51	13	3	78
PEIR Gross	5	2	3	0	10
ROCO	1	2	2	0	5
COV-CTR	2	6	1	0	9
CX-CHR	1	1	1	3	6

DOI: 10.7717/peerj-cs.3474/table-10

Note:

The data highlights a clear trend: Transformer-based models dominate across datasets, particularly large ones like MIMIC-CXR. LLMs are a rapidly emerging direction, while rule-driven methods are sparsely used.

Other datasets

Beyond the fundamental role of benchmark datasets in AMRG research, the construction and utilization of diverse and multidimensional datasets hold irreplaceable strategic value in advancing this field. While benchmark datasets provide a standardized platform for the initial validation of algorithmic models, they possess inherent limitations such as single-modality data, restricted case diversity, and idealized clinical settings. Consequently, models trained solely on these datasets often struggle to adapt to the complexities of real-world medical environments. To overcome these limitations, the development of dedicated datasets tailored to specific clinical needs, technical challenges, and application scenarios has become increasingly crucial. These specialized datasets not only complement benchmark datasets but also drive medical image report generation technology toward greater clinical applicability. In this section, we introduce several additional datasets and explore their potential applications in AMRG tasks. Further details can be found in Table 11.

INbreast dataset (Moreira et al., 2012): The INbreast dataset is a publicly available medical imaging dataset designed for breast cancer detection and diagnosis. It was collected at the Centro Hospitalar de São João (CHSJ) Breast Centre in Porto, Portugal, with approval from the Portuguese National Data Protection Commission and the Hospital Ethics Committee. The dataset is specifically tailored for mammography research and comprises 115 cases, including 90 female patients with both breasts affected and 25 patients who had undergone unilateral mastectomy. In total, the dataset contains 410 full-field digital mammography (FFDM) images depicting various types of breast lesions. Unlike digitized mammograms, FFDM images offer higher image quality and variability, making this dataset an invaluable resource for computer-aided breast cancer diagnosis. Additionally, precise annotations are provided for further research applications.

ChestX-ray14 (Wang et al., 2017): The ChestX-ray14 dataset is an extended version of ChestX-ray8, which is released by the National Institutes of Health (NIH). ChestX-ray8 contains 108,948 X-ray images (eight chest diseases) from 32,717 patients, and ChestX-ray14 further increases the disease types and sample size with a collection of 112,120 X-ray images including 14 chest diseases (atelectasis, consolidation, infiltration, pneumothorax, edema, emphysema, fibrosis, effusion, pneumonia, pleural thickening, cardiac hypertrophy, nodules, swelling, and hernia) from 30,805 patients. Each X-ray image is assigned with the corresponding disease label, and the labels are annotated by professional radiologists based on the results of medical imaging diagnosis. The purpose of this dataset is to enable the data-hungry deep neural network paradigm to create clinically meaningful applications, including common disease pattern mining, disease correlation analysis, automatic radiology report generation, etc.

Lumbar spine (Han et al., 2018): The lumbar spine dataset consists of MRI scan images of 253 clinical patients (147 females and 106 males). Han et al. (2018) constructed this dataset and applied it to the AMRG task, attempting to automatically generate unified reports of lumbar spine MRI in the field of radiology to support clinical decision-making. They combined deep learning and symbolic program synthesis theory to build a weakly supervised framework, and the generated reports covered almost all types of lumbar structures, including six intervertebral discs, six neural foramina, and five lumbar vertebrae. Unfortunately, the dataset is not public, and the work has not published the code and ablation experiments, which poses a challenge to in-depth research.

CheXpert (Irvin et al., 2019): CheXpert is a large chest X-ray image dataset released by Stanford University, which contains 224,316 chest X-ray images of 65,240 patients. Each image is accurately annotated and covers 14 different disease categories. It is worth noting that the annotation of this dataset was performed by professional radiologists, ensuring the diagnostic accuracy of the labeled data. In addition, Irvin et al. (2019) designed a labeler to automatically detect the presence of 14 observations in radiology reports and capture the inherent uncertainty in the interpretation of X-rays. As described in ‘Clinical Efficacy (CE)’, the proposed tagger brings CE evaluation to the AMRG field.

PadChest (Bustos et al., 2020): PadChest is a labeled, large-scale, high-resolution chest X-ray dataset for automatic exploration of medical images and their associated reports. Bustos et al. (2020) collected over 160,000 images of 67,000 patients, interpreted and reported by radiologists at Hospital San Juan between 2009 and 2017. It covers six different positional views as well as additional information about image acquisition and patient demographics. Among them, 27% of the reports were annotated manually by expert doctors, while the rest were performed using a supervised method based on recurrent neural networks. It is worth noting that these reports are in Spanish. For the AMGR field, training datasets with different languages is beneficial and challenging for the generalization ability of the model. Our investigation revealed that despite the public availability of the dataset, it has seen limited adoption in research with unclear patterns of implementation across studies. However, it is undeniable that PadChest is an indispensable and important resource in the AMRG field.

Gallbladder/Kidney/Liver (Zeng et al., 2020): Zeng et al. (2020) collaborated with a hospital in Chongqing, China, to collect ultrasound images of clinical examinations from 2014 to 2015. The original dataset contained ultrasound images and reports of 25,659 patients, with each patient providing one ultrasound image and the corresponding report. The images contain three organs (gallbladder, kidney, liver) and 11 diseases (gallbladder stone, gallbladder polyp, normal gallbladder, hydronephrosis, kidney stone, renal cysts, normal kidney, liver cyst, fatty liver, hemangioma, normal liver). They applied the ultrasound dataset to a semantic fusion network (SFNet) consisting of a lesion region detection model and a diagnosis generation model to automatically generate ultrasound reports and achieved SOTA results. The model used a target detection algorithm to automatically separate lesion and background areas in medical images, then utilized the isolated lesion regions to derive coding vectors and pathological information. This approach ensured that the coding vectors contained enriched lesion data, resulting in more accurate pathological information in generated diagnostic reports. Notably, the pathological information of the generated report is extracted through keyword matching (similar to BLEU as a post-processing step), which causes the generated pathological information to be different from the actual pathological information and cannot be used for parameter optimization.

FFA-IR (Li et al., 2021): FFA-IR (Fundus Fluorescein Angiography Images and Reports) is a medical image dataset for fundus lesion research that focuses on the automated diagnosis and report generation of retinal vascular lesions. Unlike traditional fundus photographs, FFA images can visualize the flow and abnormalities of retinal blood vessels in detail by injecting fluorescein dye, which helps to identify fundus lesions such as diabetic retinopathy and macular degeneration. The FFA-IR dataset contains 10,790 bilingual Chinese-English reports and 1,048,584 FFA images from clinical practice, including interpretable annotations based on schemas of 46 lesion categories. Along with the dataset, a set of nine human evaluation criteria are proposed to assess the quality of generated reports and study the reliability of natural language generation metrics in the medical field.

GE (Cao et al., 2023): Gastrointestinal Endoscope image dataset (GE) is a private dataset containing white light images of 3,168 patients from the Department of Gastroenterology and their Chinese reports. The dataset provides multiple gastrointestinal endoscopic images per patient, acquired from different perspectives, along with their associated medical reports. Cao et al. (2023) obtained 15,345 images and 3,069 reports collected from the dataset by selecting patients with five images and collected 126 medical terminologies from gastroenterologists, including 89 abnormal findings and 37 normal findings. They constructed a Multi-modal Memory Transformer Network (MMTN), which leveraged the cross-modal complementarity of multimodal medical features to weigh the contributions of visual and language features and generated medical reports that are consistent with image reports. This method achieved advanced results and has provided directions and ideas for future research.

Ophthalmic dataset (Wang et al., 2024b): The Ophthalmic Dataset is a labeled ophthalmology dataset containing 4,858 ultrasound images of 2,417 patients and their corresponding text reports that describe the imaging manifestations and corresponding anatomical locations of 15 typical intraocular diseases. Each image shows three blood flow indices of three specific arteries, providing nine parameter values to describe the spectral characteristics of blood flow distribution. In Wang et al. (2024b), this dataset was also used to evaluate cross-modal medical report generation models, including R2Gen (Chen et al., 2020) and CMN (Chen et al., 2022) models. The authors demonstrated the effectiveness of this dataset for medical report generation through visualization and other methods. Their findings also suggested that the dataset could contribute to the development of automatic diagnostic learning algorithms in the field of ophthalmology, thus reducing the pressure on ophthalmologists in clinical work.

CT-RATE (Hamamci et al., 2024): CT-RATE is the first dataset that pairs 3D medical images with their corresponding text reports. The dataset includes 25,692 non-contrast 3D chest CT scans from 21,304 unique patients. Through various reconstructions, these scans have been expanded to 50,188 volumes, totaling more than 14.3 million 2D slices. Each scan is matched with its corresponding radiology report. The construction of CT-RATE represents a major advancement in the generation of 3D medical image reports and plays a pioneering role in research in the AMRG field.

Breast/Thyroid/Liver (Li et al., 2024): Li et al. (2024) constructed three large-scale ultrasound image text datasets from different organs: a breast dataset comprising 3,521 patients, a thyroid dataset with 2,474 patients, and a liver dataset containing 1,395 patients. Each set of ultrasound images is associated with a report. For ultrasound report generation, they proposed a framework that used an unsupervised learning method to extract latent knowledge from ultrasound text reports. This information serves as prior information to guide the model for aligning visual and text features, thereby addressing the challenge of feature differences.

HiSBreast dataset (Luong, Nguyen & Thai-Nghe, 2024): The HiSBreast dataset was collected using HiS software at Ca Mau Provincial General Hospital, Vietnam, by VNPT Group. It contains breast ultrasound images of 972 hospitalized patients who underwent breast ultrasound between 2018 and 2022. Each sample includes an ultrasound image and a description of the disease symptoms based on the image, as well as the clinical diagnosis. Luong, Nguyen & Thai-Nghe (2024) utilized this dataset alongside the INbreast Dataset for constructing a breast cancer image caption generator, and the generator achieved impressive results.

3D-BRAINCT (Li et al., 2025a): The 3D-BrainCT dataset comprises 18,885 brain CT scans (742,501 slices) and paired text-scan records from 9,689 patients. It encompasses a wide spectrum of cases, including normal brain anatomy, chronic conditions, previous infarcts with residual manifestations, and acute brain lesions. The associated reports provide detailed information regarding lesion degree, spatial landmark, visual features, and final impression, making the dataset highly valuable for fine-grained diagnostic reasoning and generation tasks. The CT images include a variety of common neurology conditions affecting the skull, brain parenchyma, nasal sinuses, and the eye, increasing its generalizability to real-world clinical scenarios.

MM-Retinal (Wu et al., 2025): The MM-Retinal dataset is a multi-modal retinal imaging dataset that includes 2,169 color fundus photography (CFP) cases, 1,947 fundus fluorescein angiography (FFA) cases, and 233 optical coherence tomography (OCT) cases. Each case is provided with an image and texts in both English and Chinese. Upon MM-Retinal, MM-Retinal V2 consists of 6,720 CFP cases, 5,119 FFA cases, and 5,502 OCT cases, and covers over 96 fundus diseases and abnormalities. The MM-Retinal V2 dataset is further enriched with a text-only subset, containing a total of 452K utterances, providing a substantial repository of textual information for multimodal retinal disease analysis and report generation. Izhar et al. (2025) used the MM-Retinal dataset to explore report generation in retinal imaging, which is another step forward in the expansion of AMRG research from chest radiographs to multimodality.

A critical discussion of dataset biases and limitations

Although benchmark datasets have significantly advanced the field of AMRG, a critical examination of them reveals several underlying biases and limitations that not only affect the performance evaluation of current models but also potentially hinder the translation of technology to real-world clinical settings.

The current AMRG research landscape suffers from significant modality bias. Most of the large, high-impact benchmark datasets analyzed in this review, such as IU-Xray, MIMIC-CXR, and CX-CHR, consist entirely of chest X-rays (CXRs). While CXRs are one of the most common imaging examinations, this overrepresentation leads to the following issues: First, the vast majority of SOTA models are trained and optimized on CXRs, leaving a significant uncertainty regarding their ability to generalize directly to modalities such as CT, MRI, or ultrasound, which have fundamentally different characteristics (e.g., 3D data and varying tissue contrast). Second, the homogeneity of datasets has led to a convergence of research directions, resulting in a relative lag in research on report generation for other equally important imaging modalities. This overrepresentation is not merely anecdotal; of the 146 studies included in this review, our own analysis (summarized in Fig. 10) confirms that studies utilizing the IU-Xray and MIMIC-CXR datasets account for the overwhelming majority of published research, starkly illustrating this modality bias.

The “gold standard” for the AMRG task, which is normally radiologist-written reports, is inherently rife with variability. Different doctors vary in their reporting styles, terminology, and level of detail, which introduces label noise into model training. In the PadChest dataset, some reports are manually annotated, while others are generated using supervised methods, which can introduce systematic label inconsistencies. This noise makes it difficult for the model to learn a stable and reliable image-text mapping and can affect the reliability of evaluation results.

Currently, the vast majority of research studies train and test models on internal partitions of a single dataset. While this practice is standard in academia, it cannot guarantee model performance on external data. A model that performs well on data from Hospital A may experience a sharp decline in performance when tested on data from Hospital B (which may use equipment from a different manufacturer, have different scanning protocols, or serve a different patient population). This widespread lack of external validation is one of the key gaps between current AMRG research and actual clinical deployment.

This review also covers several high-quality but private datasets, such as CX-CHR, lumbar spine, and GE. While these datasets have advanced research in specific areas, their closed nature poses an obstacle to the long-term development of the field as a whole. First, reliance on proprietary data fundamentally undermines research reproducibility, a cornerstone of the scientific method. When a SOTA result is produced on a dataset that is not publicly accessible, independent third-party verification becomes impossible. This makes it impossible for the academic community to confirm the robustness of the result or accurately determine whether its success stems from genuine algorithmic innovation or simply exploits undisclosed features of a specific dataset. Second, closed datasets hinder fair and standardized benchmarking. Progress in the AMRG field relies on new models consistently competing against the best existing models. If the leading baseline is established on a closed dataset, subsequent innovators cannot directly compare their performance. This not only makes it difficult to assess the true value of new technologies but also can lead to a waste of research resources, as researchers may reinvent themselves on problems that have already been solved. Finally, this practice creates research silos, slowing the collective pace of innovation across the field. Scientific progress is a collaborative and cumulative process that relies on the free flow of ideas and tools. When critical data resources are locked within a few institutions, the chain of shared knowledge is broken. This stands in stark contrast to the open science ethos championed by large public benchmarks like MIMIC-CXR, which have greatly accelerated the pace of innovation across the field by providing open, accessible resources. Therefore, while private datasets have value in exploring specific questions, they pose a significant constraint to the long-term health of the scientific community.

To promote healthier and faster development in the AMRG field, future research requires not only algorithmic innovations but also concerted efforts to build more diverse, high-quality, externally validated, and openly shared datasets.