Advancing brain tumor detection: harnessing the Swin Transformer’s power for accurate classification and performance analysis

Preprocessing steps

Data preprocessing holds immense importance in machine learning pipelines. It lays the foundation for model success by shaping raw data into a format that is conducive to effective learning. The following preprocessing steps were involved.

Swin Transformer architecture

In the Swin Transformer (Swin-T) architecture, the patch-splitting module is employed to partition the RGB input image into non-overlapping patches. Each patch is treated as a “token”, and its properties are formed by merging or combining the pixels of the input RGB. This approach utilizes 4 × 4 dimensions, resulting in a feature dimension of 48 (4  ×  4  ×  3). Subsequently, a linear embedding process is applied to the features, mapping them to a user-defined size denoted as C. Multiple blocks within Swin-T incorporate a customized self-attention calculation, which is then applied to these tokens. The amalgamation of these blocks and the embedded system constitutes what is termed “stage 1”. This stage ensures the uniformity of token count, specifically H * 4 × W * 4.

 
 
 Algorithm 1: Swin Transformer Image Classification 
    Data: Input data 
    Result: Output data 
    Import necessary libraries and packages 
    Function Data_Preparation 
      Define class names class_names ← [’glioma’, ’meningioma’, ’no tumor’, ’pituitary’] 
   class_names_label ← CreateDictionary(class_names) 
   Define dataset directories tumor_directory ← ”path/to/tumor_directory” 
   for directory in [class names] do 
       file_list ← GetFilesInDirectory(directory) for file in file_list do 
   Append file to filepaths if directory is glioma then 
   Append ’glioma_tumor’ to labels 
   else if directory is meningioma then 
   Append ’meningioma_tumor’ to labels 
   else if directory is pituitary then 
   Append ’pituitary_tumor’ to labels 
   else 
   Append ’no_tumor’ to labels 
   Created DataFrame 
    Function Swin_Transformer_Model_Setup 
         Hyperparameters and settings patch_size ← (2, 2) 
   dropout_rate ← 0.03 
   num_heads ← 8 
   embed_dim ← 64 
   window_size ← 2 
   shift_size ← 1 
   num_mlp ← 256 
   num_patch_x ← input_shape[0] / patch_size[0] 
   num_patch_y ← input_shape[1] / patch_size[1] 
   learning_rate ← 1e-3 
   batch_size ← 128 
   num_epochs ← 10 
   weight_decay ← 0.0001 
   label_smoothing ← 0.1 
   Function window_partition 
   Function window_reverse 
   Function DropPath 
   Function WindowAttention 
   Function PatchExtract 
   Function PatchEmbedding 
   Function PatchMerging 
    Main 
    Data_Preparation Swin_Transformer_Model_Setup    

As the network’s depth increases, patch merging layers are employed to progressively reduce the number of tokens, resulting in a hierarchical arrangement. Initially, the patch merging layer combines the attributes of neighboring patches in a 2 × 2 group. This is followed by a linear layer operating on the concatenated 4C-dimensional features, effectively downsampling the token count by a factor of 2 × 2, equivalent to a 2× down-sampling of resolution. The resulting output size is 2C. After the downsampling in “Stage 1”, Swin Transformer blocks are employed to transform the features while preserving the resolution at H * 8 × W * 8. This stage, encompassing the initial patch merging and feature transformation, is denoted as “Stage 2”. These steps are then reiterated twice more, known as “Stage 3” and “Stage 4”, resulting in output resolutions of H * 16 × W * 16 and H * 32 × W * 32, respectively. By integrating these stages, a hierarchical representation is constructed with feature map resolutions comparable to conventional methods. As a result, the proposed architecture possesses the potential to function as a versatile alternative to the spine systems in existing approaches, yielding enhanced performance across a spectrum of vision objectives.

Block of Swin Transformer

The Swin-T architecture presents an alteration to the conventional Transformer block by incorporating a Multi-Head Self-Attention (MSA) module founded on a shifted window approach. While the other layers within the Swin-T block remain unaltered, Fig. 2 illustrates. Specifically, the Swin Transformer block is structured with a shifted window-based MSA module, succeeded by a 2-layer multi-layer perceptron (MLP) with GELU non-linearity situated in-between. Layer normalization (LN) layers are applied to precede each MSA module and each MLP layer, and residual connections are introduced after each module to guarantee the preservation of information flow.

Non-overlapping windows for self-attention

To address the computational inefficiencies of global self-attention, self-attention within localized windows has been used. These windows are organized in a non-overlapping fashion to achieve an even partition of the image. Each window is composed of M × M patches, with a typical default value of M being 7. By adopting this window-based self-attention, the computational complexity transforms into a linear relationship with the number of patches, h × w, in contrast to the quadratic complexity associated with global self-attention. This transition to a window-based approach enhances scalability and affordability, particularly in scenarios involving a substantial number of patches (h × w), where the computational demands of global self-attention become infeasible.

Shifted window partitioning approach

To address the constraint of the window-based self-attention module, which does not establish connections across windows, an innovative solution known as the shifted window partitioning technique is used. This novel approach brings in cross-window connections while upholding the efficiency of non-overlapping windows. In successive Swin Transformer blocks, we adopt two distinct partitioning strategies alternately. The first module utilizes a regular window partitioning strategy, dividing the initial 8 × 8 feature map into 2 × 2 windows of size 4 × 4 (M = 4). In the subsequent module, a shifted windowing configuration is adopted, where the windows are shifted from the previous layer’s configuration. This alternating approach enhances the connectivity between windows and enables more comprehensive information exchange throughout the network. The displacement is achieved by moving the windows by (bM/2, bM/2) pixels from the regularly partitioned windows. This shifted window partitioning approach allows for the computation of consecutive Swin Transformer blocks while incorporating cross-window connections. It enhances the modeling power of the architecture while maintaining efficient computations. The internal connection of the mathematical working of Swin-T: (1)${\displaystyle z^l=\text{MLP}\left(\text{LN}\left({\hat{z}}^l\right)\right)+{\hat{z}}^l}$ (2)${\displaystyle z^{l+1}=\text{MLP}\left(\text{LN}\left({\hat{z}}^{l+1}\right)\right)+{\hat{z}}^{l+1}}$ (3)${\displaystyle {\hat{z}}^l=\text{W-MSA}\left(\text{LN}\left(z^{l-1}\right)\right)+z^{l-1}}$ (4)${\displaystyle {\hat{z}}^{l+1}=\text{SW-MSA}\left(\text{LN}\left(z^l\right)\right)+z^l}$

The standard Window-based, multi-head, and self-attention module (W-MSA) is a major part of the Swin Transformer block. The LN is present in the front, whereas S(W)-MSA is situated at the back. There are two GELU non-linearities in the last part of the block, that is, MPL. Due to this reason, this transformer has a multiple of two. In the above equations, the output of f (S) W-MSA is indicated as ${\hat{z}}^l$, that of MPL is z^l whereas l is the position of the block of the Swin Transformer.

Self-attention, multi-head and Window-based module

W-MSA first applies linear transformations to the input features, separating them into query, key, and value components. The attention scores are calculated by multiplying the query and key tensors and scaled by the square root of the head dimension. To introduce cross-window connections, a relative position bias is computed and added to the attention scores. This bias captures the relationships between different positions within the local windows. Additionally, if a mask is provided, it is incorporated into the attention scores to handle masked tokens. The attention scores are then softmax along the last dimension, followed by applying dropout. Finally, the value tensor is multiplied by the attention scores, and the resulting tensor is reshaped, projected, and subjected to dropout before being returned as the output of the self-attention and window-based module. Overall, the multi-head functionality is implicitly included in the Window Attention implementation. The attention scores are calculated independently for each attention head, allowing the model to capture different aspects of the input information. This helps enhance the model’s representational capacity and allows for capturing diverse patterns and dependencies in the data.

To evaluate the model’s performance, several parameters were used, which play a crucial role in classification algorithms: true positive (TP), true negative (TN), false negative (FN), and false positive (FP). These metrics are essential in assessing how well the model is performing in terms of correctly identifying tumor classes.

• True positive rate (TPR): The percentage of real positive cases that a classification model correctly classifies as positive. (5)${\displaystyle \text{TPR}=\frac{\text{TP}}{\text{TP}+\text{FN}}}$

• True negative rate (TNR): It signifies the proportion of negative instances correctly classified as negative by a classification model. (6)${\displaystyle \text{(TNR)}=\frac{\text{TN}}{\text{TN}+\text{FP}}}$

• Positive predictive value (PPV): It denotes the proportion of successfully predicted positive cases to all instances that a model accurately believed to be positive. (7)${\displaystyle \text{PPV}=\frac{\text{TP}}{\text{TP}+\text{FP}}}$

• Negative predictive value (NPV): It signifies the percentage of correctly predicted negative cases among all instances that a model predicts to be negative. (8)${\displaystyle \text{NPV}=\frac{\text{TN}}{\text{TN}+\text{FN}}}$

• False negative rate (FNR): It represents the ratio of actual positive instances incorrectly classified as negative by a model. (9)${\displaystyle \text{FNR}=\frac{\text{FN}}{\text{FN}+\text{TP}}}$

• False positive rate (FPR): It indicates the ratio of genuine negative instances incorrectly predicted as positive by a model. (10)${\displaystyle \text{FPR}=\frac{\text{FP}}{\text{FP}+\text{TN}}}$

• False discovery rate (FDR): It represents the percentage of expected positive cases that are not positive. (11)${\displaystyle \text{FDR}=\frac{\text{FP}}{\text{FP}+\text{TP}}}$

• False omission rate (FOR): It signifies the percentage of real negative events that are mistakenly forecast as positive. (12)${\displaystyle \text{FOR}=\frac{\text{FN}}{\text{FN}+\text{TN}}}$

• Positive likelihood ratio (LdRo+): It is the link between the likelihood of a positive test result under specific conditions and the likelihood of a positive test result under different conditions. (13)${\displaystyle \text{LdRo+}=\frac{\text{TPR}}{\text{FPR}}}$

• Negative likelihood ratio (LdRo-): It is the ratio of the likelihood that a test result will be negative given the absence of a condition to the likelihood that the condition will be present. (14)${\displaystyle \text{LdRo-}=\frac{\text{FNR}}{\text{TNR}}}$

• Prevalence threshold (PT): It is the predictive probability level at which the positive predictive value matches the negative predictive value. (15)${\displaystyle \text{PT}=\frac{\sqrt{\text{FPR}}}{\sqrt{\text{TPR}}+\sqrt{\text{FPR}}}}$

• Threat score (TS): It is the product of the True positive rate and the positive predictive value, encapsulating the combined accuracy of positive predictions. (16)${\displaystyle \text{TS}=\frac{\text{TP}}{\text{TP}+\text{FN}+\text{FP}}}$

• Prevalence (Pe): It pertains to the proportion of the population (P) afflicted by the condition (N) under study. (17)${\displaystyle \text{Pe}=\frac{\text{P}}{\text{P}+\text{N}}}$

• Accuracy (AC): It represents the proportion of correct predictions made by a model across all instances. (18)${\displaystyle \text{AC}=\frac{\text{TP}+\text{TN}}{\text{TP}+\text{TN}+\text{FP}+\text{FN}}}$

• Balanced accuracy (BA): It is the mathematical mean of the real positive rates and real negative rates, offering a holistic view of model performance. (19)${\displaystyle \text{BA}=\frac{\text{TPR}+\text{TNR}}{2}}$

• F1 score (F1): It is the harmonic mean of the true positive rate and the positive predictive value, balancing precision and recall. (20)${\displaystyle \text{F1}=\frac{2\times \text{TP}}{2\times \text{TP}+\text{FP}+\text{FN}}}$

• Matthews correlation coefficient (MCC): It gauges the effectiveness of a binary classification model, considering true and false positives and negatives. (21)${\displaystyle \text{MCC}=\frac{\text{TP}\times \text{TN}-\text{FP}\times \text{FN}}{\sqrt{\left(\text{TP}+\text{FP}\right)\left(\text{TP}+\text{FN}\right)\left(\text{TN}+\text{FP}\right)\left(\text{TN}+\text{FN}\right)}}}$

• Fowlkes–Mallows Index (FMI): It quantifies the resemblance between observed and predicted classifications. (22)${\displaystyle \text{FMI}=\sqrt{\text{TPR}\times \text{PPV}}}$

• Informedness (BM): It is the disparity between the true positive rate and the false positive rate, providing insight into classification model performance. (23)${\displaystyle \text{BM}=\text{TPR}+\text{TNR}-1}$

• Markedness (MK): It signifies the difference between the positive predictive value and the negative predictive value, reflecting the model’s predictive accuracy. (24)${\displaystyle \text{MK}=\text{PPV}+\text{NPV}-1}$

• Diagnostic odds ratio (DOR): It is the ratio of the positive likelihood ratio to the negative likelihood ratio, capturing the diagnostic power of a test. (25)${\displaystyle \text{DOR}=\frac{\text{LdRo+}}{\text{LdRo-}}}$

Visual depiction of model’s performance

The model’s accuracy underwent comprehensive assessment and visualization via a training and validation accuracy graph depicted in Fig. 3. Within this graph, the blue line denotes the accuracy progression of the training set, while the orange line signifies the model’s accuracy concerning the validation set. Initially, the training accuracy commenced slightly above 0.90 on the vertical axis and showed a gradual ascent across epochs. It eventually reached a stable phase marginally below 0.98, suggesting a consistent training pattern where the model’s accuracy improved and then plateaued. Conversely, the orange line, representing validation accuracy, commenced at 0.96 on the y-axis. It demonstrated a more variable trajectory characterized by fluctuations, signifying the model’s varying efficiency on the validation set during training. These fluctuations showcased moments of higher accuracy interspersed with periods of comparatively lower accuracy.

Figure 4 illustrates the model’s loss dynamics. The blue line tracks the training loss, while the orange line showcases the validation loss. Initially, the training loss starts at its highest value on the y-axis and progressively diminishes across epochs, steadily converging towards a specific value. Around a point marginally below 0.4, the training loss levels off, forming a horizontal plateau parallel to the x-axis, denoting convergence. Contrastingly, the orange line, denoting validation loss, initiates at 0.47. It portrays a more fluctuating pattern, characterized by peaks, signifying fluctuations in the model’s performance on the validation set throughout the training process.

The confusion matrix illustrates the model’s performance in classifying different tumor types, highlighting strengths and areas needing improvement. It showcases the model’s ability to differentiate between various classes while indicating potential focus areas for enhancing classification accuracy, especially in classes with higher false predictions. In Fig. 5, the model shows robust performance in correctly identifying glioma and pituitary tumors, with very few false predictions. Meningioma and “no tumor” classes have slightly higher false predictions, indicating some misclassification. Overall, there are minimal false negatives (FN) across all classes, suggesting the model’s sensitivity in detecting actual positive instances.

Correctly classified images: Fig. 6 showcasing sample images correctly classified by the model adds credibility to its accuracy. Highlighting these instances demonstrates the model’s capability to accurately identify distinct tumor types. Incorrectly classified images: Fig. 7 presents images that the model misclassified and provides valuable insights into its limitations. Understanding where the model faltered can guide improvements or adjustments in future iterations. By incorporating visual examples, we are not just presenting numbers but providing real-world instances that enhance the audience’s understanding of the model’s capabilities and potential areas for enhancement.

In Table 3, we evaluate the effectiveness of our proposed Swim Transformer algorithm in the context of brain tumor classification, alongside several established methods. The metrics under consideration primarily revolve around accuracy, a crucial factor in medical image analysis. Swin Transformer exhibits remarkable performance, achieving an accuracy of 97%, outperforming the other methods presented in the table. Notably, the Swin Transformer surpasses the accuracy of CNN, Deep-CNN, Fused-based ensemble methods, deep neural networks, CNN and NADE, ViT model, ensemble CNN, BMO-ViT, and Ensemble XG-Ada-RF model by a significant margin. The distinguishing factor lies in the innovative training techniques and model adaptations incorporated in our implementation, enabling superior performance compared to prior studies. Specifically, our model’s unique approach to feature extraction and inter-window connectivity fosters heightened accuracy in discerning intricate patterns within brain tumor images. This outcome underscores the Swin Transformer’s potential as a formidable tool, promising substantial advancements in brain tumor detection accuracy critical for precise medical diagnostics and improved patient care. By showcasing superior accuracy and reliability compared to existing methods, our research demonstrates the distinctive contributions and advancements brought forth by our Swin-Transformer implementation.