Recurrent attention U-Net for segmentation and quantification of breast arterial calcifications on synthesized 2D mammograms

Contributions

In this article, we present the results of a novel recurrent attention U-Net to extract BAC structures from mammographic images. Our work was inspired by the success of using attention mechanisms to segment curvilinear structures from different imaging modalities (Fu et al., 2019; Mou et al., 2021). Moreover, the success of combining a recurrent neural network within the U-Net architecture for vessel segmentation by Alom et al. (2018). The contributions of this work are as follows:

1. We developed a new model for BAC segmentation and quantification, which first generates BAC masks used later to calculate five metrics that indicate the severity level of BAC in the segmentation mask.

2. The new model extended U-Net using recurrent blocks and an attention module to ensure better feature representation and improve the abilities of the U-Net to effectively capture long-range dependencies.

3. We published a new dataset with the ground-truth of BAC, which was annotated by radiologists. This dataset with the annotations will be made available to foster future research in this area.

The article is structured as follows: “Introduction” introduces the method. “Related Work” reviews related work, highlighting significant contributions and identifying gaps in the current literature on DL networks for BAC segmentation. The DL networks for our BAC segmentation model are presented in “Approach”. The datasets, quantification metrics, and experimental setup are presented in “Experiments”. The experimental results are presented in “Results”. The results are discussed in “Discussion”. The conclusions are in “Conclusions”, and the promising future works are in “Promising Future Works”.

Recurrent attention U-Net

This article identifies the BAC segmentation problem as a binary semantic segmentation task, wherein each pixel is categorized into one of two classes. The segmentation task for BAC involved the classification of two different classes: BAC pixels and non-BAC pixels. To address this, we propose an end-to-end approach that is illustrated in Fig. 1. This approach systematically encompasses the entire process, from raw data preparation to the final quantification of BAC. Based on the U-Net (Ronneberger, Fischer & Brox, 2015), R2U-Net (Alom et al., 2018), and attention module (Fu et al., 2019; Mou et al., 2021), we propose a recurrent attention U-Net (RAU-Net) for BAC segmentation. The encoder and decoder modules included multiple blocks, each employing a recurrent residual block. An attention module was added between the encoder and the decoder. In order to address the issue of information loss resulting from the use of max-pooling operations, we used a skip connection between every block of the encoder and decoder, similar to the U-Net network. To obtain the final segmentation map, we applied a $1\times 1$ convolutional layer to the decoder output at the end of the model. In the case of using a different loss function than BCEWithLogitsLoss, we added the sigmoid layer to the output. A detailed structure of our network is shown in Fig. 2 and Table 1.

Recurrent block

The operations of the recurrent convolutional layers were executed in the manner defined by Alom et al. (2018). Suppose the $x$ input sample in the $l$’th layer and the center pixel of a patch placed at $(i,j)$ in input on sample the $k$’th feature map in the layers, the output of the network $O_{ijk}^l$(t) is at the time step $t$. The output of the recurrent block is expressed as Eq. (1). $x_l^{f(i,j)}(t)$ and $x_l^{r(i,j)}(t-1)$ are the inputs for the standard convolutional layers and the $l$’th recurrent convolutional layer. The $(W_k^f)$ and $(W_k^r)$ values are the weights for the standard convolutional layer and the recurrent convolutional layer of the $k$’th feature map, and the $b_k$ is the bias. The ReLU activation function $f$ received the outputs of the recurrent convolutional layer and are expressed as in Eq. (2). $\mathcal{F}\left(x_l,w_l\right)$ defines the outputs from $l$’th recurrent layer, which was used for down-sampling and up-sampling layers in the convolutional encoding and decoding model. In the case of the recurrent residual block, the output is passed to the residual of the unit, as illustrated in Fig. 3. The block’s output can be computed as in Eq. (1). $x_l$ is the block’s input samples. The $x_{l+1}$ in the R2U-Net model’s encoder and decoder units represent the input for the immediately following sub-sampling or up-sampling layers, and it can be calculated as Eq. (3).

(1) $$O_{ijk}^l(t)={\left(w_k^f\right)}^T\times x_l^{f(i,j)}(t)+{\left(w_k^r\right)}^T\times x_l^{r(i,j)}(t-1)+b_k$$

(2) $$\mathcal{F}\left(x_l,w_l\right)=f\left[O_{ijk}^l(t)\right]=max\left[0,O_{ijk}^l(t)\right]$$

(3) $$x_{l+1}=x_l+\mathcal{F}\left(x_l,w_l\right)$$

Collecting features across various time steps enhances the quality and robustness of feature representation. This helps extract very low-level features crucial for segmentation tasks, such as the tubular structure of the vessels.

Attention block

Expressive features that are attentive to channel and spatial information were generated by combining a channel attention block and a spatial attention block. The spatial attention block uses a selective aggregation mechanism to combine features at each spatial location by giving weights to the features across all spatial locations. This lets the model accurately capture how the data and features are related over long distances, even when they are far apart. In contrast, the channel attention block ensures that the entire space is used for representing and normalizing; hence, enhancing the incorporation of contrasting features across separate channels enhances the model’s ability to discriminate. Figure 4 shows the detailed structure of an attention module where, in spatial attention, after the input features F, two types of layers, $3\times 1$ and $1\times 3$ convolutional layers, are utilized to build two new feature maps $Q_y$ and $K_x$. Applying a softmax layer to the matrix multiplication of the transpose of the features captured in the vertical and horizontal directions yielded an intra-class spatial association as Eq. (4), where ${\mathcal{S}}_{(x,y)}$ denotes the $y$ position’s effect on the $x$ position. The feature correlation matrix between any two points was computed and outputted by matrix multiplication; two similar spatial points increased one another, while two dissimilar spatial points decreased one another. This operation enables the network to optimally use and learn about the structure of various spatial locations. Subsequently, the softmax function was used on the correlation matrix in order to produce an attention map that represents the degree of similarity between each spatial position and the rest. Stronger similarity increases the response between two points. A channel attention map was created by adding a softmax layer between the input feature and transposing it on the channel-wise similarity map as Eq. (5). As a result, by conducting matrix multiplication, we obtained the channel dependence matrix. The channel dependency matrix is then applied to a softmax to improve the discrimination between the BAC structure and its background. These operations improved the expressiveness of class-dependent features by enhancing the contrast between them.

(4) $${\mathcal{S}}_{(x,y)}=\frac{\exp \left(Q_y^{\mathrm{T}}\cdot K_x\right)}{{\sum }_{x^{\mathrm{\prime }}=1}^N\exp \left(Q_y^{\mathrm{T}}\cdot K_{x^{\mathrm{\prime }}}\right)}$$

(5) $${\mathcal{C}}_{(x,y)}=\frac{\exp \left(F_x\cdot F_y^{\mathrm{T}}\right)}{{\sum }_{x^{\mathrm{\prime }}=1}^C\exp \left(F_{x^{\mathrm{\prime }}}\cdot F_y^{\mathrm{T}}\right)},$$

Loss function

One of the most difficult problems encountered was the high-class imbalance between BAC foreground pixels and background pixels. The loss in the BAC segmentation task was computed from both the background and the BAC. Overall, the BAC pixels represented approximately 0.2% of the total pixels. To mitigate this issue, we experimented with various loss functions and optimizers. We tested several loss functions (MSE loss, cross entropy loss, dice loss), which showed good performance in other DL scenarios, but did not yield good results. Equation (6) shows the formula for the binary cross entropy (BCE) loss function. According to the equation, T represents labels of a single image used as ground-truth, while $T_x$ represents a single element of T. The network’s output prediction mask is shown by $P_x$. When using the BCE loss, the model tends to classify pixels into the unchanged type, which affects the result. To solve the data imbalance problem, we used BCEWithLogitsLoss, which worked well with imbalanced data because it gives each class the same amount of weight as if it had an equal portion of the dataset by the (pos weight) parameter (Xiong et al., 2021) Eq. (7). BCEWithLogitsLoss combines a sigmoid layer and the BCE loss. This approach was more numerically stable than using a plain sigmoid, followed by a BCE loss because it uses the log-sum-exp method for numerical stability by merging the operations into one layer. Adam was found to be the best optimizer for BCEWithLogitsLoss.

(6) $${\mathrm{L}}_{BCE}={\sum }_x-\left(T_x\log \left(P_x\right)+\left(1-T_x\right)\log \left(1-P_x\right)\right)$$

(7) $$\begin{array}{l}\ell (x,y)=L={\left\{l_1,\dots ,l_N\right\}}^{\top },\\ l_n=-w_n\left[t_n\cdot \log \sigma \left(x_n\right)+\left(1-t_n\right)\cdot \log \left(1-\sigma \left(x_n\right)\right)\right]\text{}{\text{}}^{\cdot }\end{array}$$

Dataset

The University of Miami Institutional Review Board (IRB) approved this research study under ID number 20191227. The IRB also approved a waiver of consent and a HIPAA full waiver of authorization. Institutional guidelines were followed for the de-identification, extraction, and storage of 2,000 craniocaudal (CC) and mediolateral oblique (MLO) synthetic views of DBT exams performed in women aged 35 years and older, specifically over the time period from January 1, 2016, to December 31, 2018. The DBT exams were performed on six hologic dimensions mammography units located at three different breast imaging centers within the University of Miami Health System (UHealth), Miami, Florida. The DICOM data from UHealth were converted into 8-bit portable network graphics (PNG) format images. Following this, an expert radiologist specialized in breast imaging, who received fellowship training and had 10 years of experience, reviewed all the images and discarded images with poor quality due to artifact and those that were not CC or MLO synthetics views. The final dataset consisted of a total of 1,436 images. Most images were large-size with 2,457 pixels in height and 1,996 pixels in width. Each one was processed as a separate image. Figure 5 shows a flowchart of the dataset pre-processing steps.

The polygon tool of Supervisely (https://supervisely.com/) was used to manually draw BAC boundaries, as shown in Fig. 6. Following dedicated training, a medical student in their third year, a resident specializing in radiology in their second year, a breast imaging fellow, and an expert fellowship-trained breast imaging radiologist with 10 years of experience, each separately performed the BAC segmentation. After completing the segmentations, the expert breast imaging radiologist evaluated each one and made any necessary modifications, to achieve gold-standard annotation. At this step, these BAC tracings were considered the ground-truth. After dividing the images into three separate non-overlapping groups for our study 1,149 (80%) images for model training, 143 (10%) for validation, and 144 (10%) for testing, we meticulously analyzed various aspects of these datasets to gain deeper insights into the BAC class distribution and area. The findings from our analysis, including the total number of images, the distribution of images with and without the BAC, as well as the average BAC area to background area ratio, and the overall average BAC area per image, are comprehensively detailed in Table 2. The dataset and ground-truth have been made publicly available on GitHub (https://github.com/Manar-ibr/BacSeg/releases/tag/0.1.0).

Evaluation criteria

The segmentation results were reported and evaluated through various widely used metrics, defined as follows:

(8) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}=\frac{\mathrm{T}\mathrm{P}+\mathrm{T}\mathrm{N}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{T}\mathrm{N}+\mathrm{F}\mathrm{N}}$$

(9) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}}$$

(10) $$\mathrm{F}1-\mathrm{s}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=\frac{2\cdot \mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\cdot \mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{p}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}$$

(11) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}}$$

(12) $$\mathrm{J}\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{a}\mathrm{r}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{d}\mathrm{e}\mathrm{x}=\frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}+\mathrm{F}\mathrm{N}}$$the terms TP, TN, FP, and FN represent the true positive, true negative, false positive, and false negative, respectively. The values were computed based on the pixel number. Nonetheless, when analyzing imbalanced datasets, accuracy can be deceiving, as it can be impacted by the classification results obtained for data that belongs to the majority class, which makes it more difficult for a classifier to perform well with data belonging to another class. For this reason, we used several metrics to deal with imbalanced datasets, including F1-score and Jaccard. Utilizing these metrics allows for a more comprehensive evaluation of a classifier’s performance on imbalanced datasets. They can help identify models that not only perform well on the majority class but also maintain reasonable performance levels on the minority BAC class. The F1-score and Jaccard index are metrics used to calculate the ratio of TP pixels to the total number of detected pixels and were used to evaluate the overlapping between the predicted mask and ground-truth.

Experiment set-up

To demonstrate the performance of the RAU-Net model, we tested it on the mammogram dataset. For this implementation, Pytorch v1.12.1 frameworks were used. Because of the imbalance between the foreground and background, the binary cross entropy loss converges slowly. Based on the experiment results, we used BCEWithLogitsLoss. The models were developed using Google colab $\mathrm{p}\mathrm{r}\mathrm{o}+$, Python 3 Google Compute Engine backend (GPU) RAM: 51.01 GB Disk: 156.99 GB. The training and testing dataset images were reshaped into $640\times 640$ to meet the requirements of the models. We enhance image contrast to emphasize the difference between the breast background tissue and the calcified vessel. We normalize the input image to reduce the impact of variability in contrast between classified vessels and background, which improves model performance and makes it converge much faster. The dataset was normalized by subtracting the mean $\mu =0.3$ and dividing by the standard deviation $\sigma =0.2$. Our model was trained from scratch using the Adam optimization method, with $\beta 1$ = 0.5 and $\beta 2$ = 0.999. The batch size was set to 1, and the learning rate (lr) was initialized at 0.0001 and decayed after 20 epochs. Each model was trained with 70 epochs. At each epoch in the validation, we calculated the model score, which was the sum of the Jaccard index and dice coefficient; if the score was better than the previous score, then the weights were updated. Consequently, the best model based on the validation set was saved. The code of RAU-Net model has been made publicly available on GitHub (https://github.com/Manar-ibr/BacSeg.git).

Ablation study

Because all previous methods’ datasets were private, comparing them to an existing method is not feasible. Therefore, we defined three different comparisons to evaluate the performance of RAU-Net: the original U-Net, the U-Net with recurrent and residual to evaluate the effectiveness of the recurrent residual block, and the U-Net with Attention to evaluate the effectiveness of the attention module.

These models were utilized in the comparative process:

• U-Net (Ronneberger, Fischer & Brox, 2015): the original model, which includes long-skip connections.

• R2U-Net (Alom et al., 2018): the U-Net with recurrent residual blocks.

• U-Net (Ronneberger, Fischer & Brox, 2015) + Attention block (AttU-Net): the U-Net with attention block between the encoder and the decoder.

The same set-up as in our model was used for U-Net, R2U-Net, and AttU-Net.

Other analysis

The results derived from the 1,436 mammography images, as described in “Dataset”, in terms of accuracy, sensitivity, F1-score, precision, and Jaccard values are presented as percentages in Table 3. As shown, the RAU-Net substantially outperformed the other models (as described in “Ablation Study”), with an overall accuracy of 99.8861%, a sensitivity of 69.6107%, a precision of 68.3948%, an F-1 score of 66.5758%, and Jaccard index of 59.5498%. The model achieved high overall accuracy, reflecting its effectiveness in correctly classifying the majority of pixels within mammograms. When compared to relevant models in the field, our model demonstrated promising results in terms of sensitivity, F-1 score, and Jaccard coefficient. These metrics, indicative of the model’s ability to accurately identify true positives BAC and the precision and overlap between the predicted segmentation and ground truth, highlight the model’s competitive performance despite the inherent challenges of the dataset. These challenges include imbalanced class distribution and the complex presentation of BAC. While the result achieved across these metrics emphasizes the model’s good capabilities in BAC segmentation, there are areas for improvement. Enhancing the sensitivity, for example, could be achieved by increasing the dataset size or employing more advanced data augmentation techniques. We compared the performance of the proposed RAU-Net using different loss functions (cross entropy loss, MSE loss, Dice loss, and BCEWithLogitsLoss). Table 4 shows that BCEWithLogitsLoss improved the segmentation results for F1-score, Jaccard index, precision, and sensitivity.

Figure 7 shows examples of BAC segmentation outputs using the RAU-Net model on mammography images. Examples are shown in Fig. 7 first column (A) and (D) where the RAU-Net correctly detects BAC comparable to human specialists. Figure 7 second column (B) and (E) displays a few mis-detected BAC results that the model incorrectly detects as BAC. Small clusters of non-continuous BAC can cause this. Figure 7 third column (C) and (F) illustrates how BAC was incorrectly detected as background in a few mammogram images. There is a possibility that this kind of error was made in the mammograms of individuals who had dense breast tissue. Dense breast tissue and the presence of clustered calcifications pose significant challenges for automated models like RAU-Net in distinguishing between normal anatomy and BAC. Dense breasts, characterized by a high proportion of fibrous and glandular tissue, can obscure mammographic details, making it difficult for automated systems to interpret the images accurately. If there are a sufficient number of situations like this for training, it is possible to prevent making this mistake. Figure 8 includes samples from the RAU-Net, U-Net, AttU-Net, and R2U-Net models results on our test dataset.

Quantification metrics

The general evaluation metrics used in segmentation help evaluate the performance of the segmentation model by performing a pixel-to-pixel evaluation. This article aimed to quantify the amount of BAC within a mammogram into a single number that can be used to define BAC severity. To do this, we used the metrics by Guo et al. (2021), and we computed the correlation for five metrics using RAU-Net predicted mask compared to the same metrics that were produced based on the ground-truth mask. Table 5 presents a summary of the R2-correlation value for the five metrics. On the 144 test images, RAU-Net had the largest R2-correlation value of 0.83 between the predicted mask and ground-truth achieved when using the T AMX metric with threshold = 100, as in the scatterplot shown in Fig. 9. These findings coincide with the article’s results (Guo et al., 2021). The evaluation of BAC quantification for the different stages of BAC is depicted in Table 6, in which the first stage includes no BAC, the second stage denotes minimal BAC, the third stage represents moderate BAC and the fourth stage for severe BAC. A higher quantification value is indicative of a more severe stage of BAC, indicating a greater risk of CVD.

ROC analysis

Figure 10 shows the ROC curves of our proposed RAU-Net, U-Net, R2U-Net, and AttU-Net over our datasets for the segmentation of BAC. It can be comparatively illustrated via the local enlarged view that the suggested model outperformed all methods for the BAC segmentation task. The area under the curve (AUC) is a popular measure of classification accuracy. In general, higher AUC values indicate better performance. AUC can be interpreted as the average true positive rate over possible false positive rates. As shown, U-Net achieved AUC = 0.59. AttU-Net presented better BAC extraction ability compared to U-Net, particularly for small BAC, and achieved AUC = 0.75. The use of the attention mechanism on high-level features in both the channel and spatial dimensions has resulted in enhanced inter-class discrimination and intra-class aggregation. R2U-Net achieved AUC = 0.87 because of better feature representation for BAC segmentation tasks. When the values were compared, the best result, 0.96 AUC, was achieved by RAU-Net, which combined the advantages of recurrent blocks to enhance feature extraction and attention module to capture the features’ long-range dependencies. This extension enhanced the result of detecting BAC, and it is a promising result in applying such a model clinically and using it for the estimation of CVD risk factors.

Feature learning

The RAU-Net model is able to recognize features and make distinctions with high accuracy. Figure 11 illustrates some examples of feature maps that were created after the recurrent block’s max-pooling layer in the contracting path. These layers were sufficiently deep to generate BAC classification features. Figure 11A demonstrates samples of learned features from non-BAC subset, whereas Fig. 11B shows samples of learned features from BAC subset. A comparison of both feature maps shows that the RAU-Net model learned the BAC samples at multiple scales and viewpoints. Moreover, the model recognized the absence of BAC in the non-BAC features and discriminated them from the other forms of calcifications seen in mammograms.

Training and testing time

Table 7 shows a comparison between RAU-Net and the three related models (as described in “Ablation Study”) in terms of training time per epoch and testing time for the test set. The fastest model was the U-Net, which was created as a segmentation model for medical images, but it does not capture the BAC after training for many epochs. AttU-Net required more time because of the computation of the attention module, which included matrix operations; in contrast, it enhanced the performance, as shown in Table 3. Adding recurrent blocks increased the computation time in the R2U-Net and RAU-Net, which was expected, given that it performed more computations in different time steps. Our RAU-Net model achieved better BAC segmentation results in a reasonable amount of time regarding both training and testing, 589 s to train one epoch and 191 s during the testing phase.