Deep learning-driven dyslexia detection model using multi-modality data

Literature Review

Research studies (Ahire et al., 2022; Jan & Khan, 2023; Marimuthu, Shivappriya & Saroja, 2021; Perera, Shiratuddin & Wong, 2018; Usman et al., 2021; Velmurugan, 2023) systematically reviewed DD techniques based on DL approaches. Spoon, Crandall & Siek (2019) used children’s handwritten images to predict dyslexia. They employed a feature extraction technique to extract the crucial patterns to differentiate between normal and abnormal individuals. In order to identify cognitive processing variations associated with dyslexia, deep learning models may use behavioral data gathered during reading activities, including response times, accuracy rates, or eye-tracking patterns (Deans et al., 2010; Nerušil et al., 2021). These models may employ long short-term memory (LSTM) and transformer architectures for sequence modeling. To better understand dyslexia, deep learning methods, including multi-modal fusion networks and attention processes, can integrate data from multiple sources (Lr & Sudha Sadasivam, 2022). Transfer learning adapts pre-trained deep learning models to extract critical features for dyslexia diagnosis. By fine-tuning these models, researchers can increase detection accuracy with limited labeled data. To supplement limited DD datasets, generative adversarial networks may provide synthetic data samples replicating genuine FMRI and EEG data (Lr & Sudha Sadasivam, 2022; Tomaz Da Silva et al., 2021; Yan, Zhou & Wong, 2022). The generalizability and resilience of deep learning models for dyslexia detection may be enhanced using GANs by augmenting the variety and volume of the training data. Zainuddin et al. (2018) proposed a model using a K-nearest neighbor algorithm based on DDM. Similarly, studies (Guhan Seshadri et al., 2023; Parmar & Paunwala, 2023; Yan, Zhou & Wong, 2022) employed EEG signals for predicting dyslexia.

Frid & Manevitz (2018) discussed the importance of feature extraction techniques in DD. Zaree, Mohebbi & Rostami (2023) proposed an ensemble learning approach integrating multiple models’ outcomes to predict dyslexia. Raatikainen et al. (2021) built a model to detect developmental dyslexia using the individual’s eye movements. Nerušil et al. (2021) developed a model using eye movements. Tomaz Da Silva et al. (2021) proposed a visualization technique to identify DD using FMRI images. A CNN model was trained with high-level features of dyslexia. The studies (Alqahtani, Alzahrani & Ramzan, 2023; Guhan Seshadri et al., 2023; Kariyawasam et al., 2019) classified multiple medical imaging techniques for DD using CNN models. These studies extracted intricate patterns from EEG signals.

Handwritten and eye movement data are valuable in DD development (Deans et al., 2010; Jothi Prabha & Bhargavi, 2022; Lou et al., 2017; Marimuthu, Shivappriya & Saroja, 2021). However, the unique writing styles, character formation, and spatial structure of handwriting can cause complexities in identifying dyslexia. The interpretation of handwriting features is based on the evaluator’s expertise. Thus, the ground truth information may vary from the annotated handwritten images. Similarly, extracting patterns from the eye movement data demands high computational resources. Reading, linguistic, cognitive, and visual processing speeds influence eye movement patterns. Eye movement patterns of dyslexic individuals may overlap with the patterns of normal individuals. Individuals with dyslexia may exhibit compensating mechanisms or adaptive techniques during reading, altering or concealing their usual eye movement patterns, making dyslexia diagnosis more challenging. The interpretability of CNN and RNN-based DDM prediction is challenging for physicians in decision-making. The complex architecture of the existing models causes difficulties in deploying them in healthcare settings. Extensive data augmentation and transfer learning techniques are required to improve the efficiency of the existing models. In addition, the current models are susceptible to overfitting due to limited datasets. Developing a DL-based DDM from scratch may demand high-performance computing resources. The effective performance of the feature extraction plays a crucial role in lowering false negative/positive results. The recent developments in CNN and RNN architectures offer an opportunity to apply them in developing DDMs. These knowledge gaps have motivated the authors to focus on multi-modality-based DD detection models.

Data acquisition

The authors acquired FMRI, MRI, and EEG data from three public repositories. Dataset 1 contained the raw 3 T MRI data and FMRI images of 58 children. The FMRI images were captured during the reading-related functional activities. Stimuli encompassing 60 words and 60 matched pseudo homophones were used to evaluate the individual’s activity. The activities were repeated three times with a short break of 3–5 min. The dataset could be accessible using the repository (https://openneuro.org/datasets/ds003126/versions/1.1.0).

Dataset 2 included the synthetic grayscale MRI images. These images were generated using the T1-weighted MRI images of 204 children. The dataset owners followed the FMRI study (Kuchinsky et al., 2012) to develop the neuroimaging data. The adaptive non-local mean algorithm was used to denoise and correct the image biases. The Gaussian Kernel method was used to minimize the false positive results. The dataset could be downloaded from the repository (Vaden et al., 2020) (https://data.mendeley.com/datasets/3w9662wjpr/1).

Dataset 3 contained the EEG data of ten college students watching videos of introductory algebra and quantum mechanics. A total of 20 videos were used to collect data from the participants. A headband, MindSet was used to measure the participants’ brain activity. The dataset covered 12,811 rows ×15 columns of EEG data, including subject ID, mediation, attention, delta, etc. The dataset was available in the Kaggle repository (https://www.kaggle.com/code/alrohit/eeg-analysis-of-confusion-for-dyslexia-diagnosis/notebook).

The datasets were available in the public repository. Researchers can use these data without any restrictions. The dataset owners obtained the participants’ consent to share the data for research purposes.

In order to generate multiple images from the FMRI images, the authors developed a 3D CNN model. The model contained five convolution layers with filter sizes of 32, 64, 128, 256, and 512, and maxpooling layers. The features were flattened. Subsequently, dense and dropout layers with rectified linear unit (ReLu) and Sigmoid activation functions were employed to generate the images. A channel dimension integration and resize functions were applied to resize the images into 224 × 224 pixels. Moreover, data augmentation techniques were employed to train the feature extraction models.

MobileNet V3-based feature extraction

The inverted residuals and architectural optimization improve the performance of the MobileNet V3 model in medical image classification and semantic segmentation. The utilization of the SE block, hard-swish activation function, and efficient last-stage convolution layers lead to better feature representation compared to the existing pre-trained models. The flexibility and adaptability of the MobileNet V3 model have motivated the authors to apply it in this study. In addition, the MobileNet V3 model was lightweight and supported the proposed DDM to detect dyslexia in a resource-constrained environment. The depthwise separable, convolutions, linear bottlenecks, and inverted residuals minimized the computational complexities. Channel-wise feature calibration was used to enhance the feature representation. However, integrating standard squeeze and excitation blocks could improve the model’s efficacy.

The authors developed a shallow CNN model with a MobileNet V3 backbone in this study. They froze the initial layers of the MobileNet V3 model to prevent overfitting and reduce the computational cost. The lightweight nature of the MobileNet V3 model may reduce the ability to extract intricate features of dyslexia. Integrating the SE mechanism can improve the MobileNet model’s performance by enhancing feature representation and discrimination power. In addition, the SE mechanism facilitates the MobileNet model in learning informative dyslexia features. Thus, the authors integrated the SE block with the backbone model. Figure 2 highlights the architecture of the recommended feature extraction using the MobileNet V3-model.

Equation 1 shows the computational form of SE integration. Introducing the SE block into the model improved the adaptive recalibration of the channel-wise features. The SE block contained a global average pooling layer, reshape function, and dense layers. (1)${\displaystyle T=MobileNetV3\text{\_}SE\left(X,F,K,E,S,R\right)}$

where T is the output tensor, X is the input tensor, F is the number of filters, K is the kernel size, E is the expansion ratio, S is the stride, and R is the squeeze ratio.

EfficientNet B7-based feature extraction

The EfficientNet B7 model was the highly effective variant of the EfficientNet models. It offered complex patterns and features by maintaining computational efficiency. Medical imaging analysis models use the EfficientNet B7 models to generate a better outcome. EfficientNet model uses bottleneck layer and depthwise separable convolutions for feature extraction. The compound scaling functionality can balance model depth, width, and resolution. The EfficientNet model can enable high-quality, efficient, and scalable solutions for real-world challenges. These features of the EfficientNet B7 model motivated the authors to employ it for extracting key features from the FMRI images. The authors built a CNN model using six convolution layers, batch normalization, and dropout layers. They used the EfficientNet B7 model’s weights to generate features. The compound scaling method was employed to maintain a trade-off between model depth and resolution. The self-attention mechanism was introduced using the global average pooling layer, reshape, and permute functions for the critical feature selection. The self-attention mechanism captures global dependencies in the multi-modality data, focusing on dyslexia-related patterns while neglecting irrelevant features. It assigns unique attention weights to different parts of the input sequences. It facilitates adaptive feature fusion across modalities. It identifies the structural differences in the brains of DIs compared to the normal individuals. These differences may encompass modifications in the size and asymmetry of brain regions related to language processing. The patterns associated with fractional anisotropy were extracted to assist the proposed model in producing optimal accuracy. Equation 2 shows the computation of the attention weights using a scaled dot-product attention approach. (2)${\displaystyle attentionweights=Softmax\left(Mat\text{\_}Mul\left(query,key,transpose\right)\right)}$

where the Mat_Mul function performs matrix multiplication of query and key tensors for the critical feature identification, transpose indicates the transposition of crucial tensor, and the Softmax function computes the exponential of each element of the matrix and normalizes it using the sum of all elements.

Bi-LSTM-based feature extraction

The motivation to apply the Bi-LSTM model to extract features from EEG data stemmed from the bi-directional architecture. Bi-LSTM can capture temporal dependencies in sequential data. It can learn from past performance and tune itself for future direction. EEG data are susceptible to artifacts from eye movements, muscle activity, and external interference. Bi-LSTM can mitigate the effect of noise on feature extraction. To enhance the performance of the Bi-LSTM, the authors introduced a self-attention mechanism and early stopping strategies. In the context of EEG feature extraction, the self attention mechanism focuses on the mismatched negativity response, reflecting the brain’s ability to identify and process auditory stimuli. The connectivity patterns among auditory, language, and reading-related brain regions were detected in order to predict dyslexia using EEG. The structure of Bi-LSTM-based feature extraction is highlighted in Fig. 3. Equation 3 shows the mathematical form of the Bi-LSTM layer. (3)${\displaystyle P=Model.add\left(Bi-LSTM\left(Units=64,retur{n}_{sequence}=True\right),inpu{t}_{shape}=T,C\right)}$

where P is the prediction, Units specify the number of neurons, return_sequence represents the sequence of outputs for each timestep, input_shape shows the shape of input with time step for each sequence, and channels show the number of features.

The self-attention mechanism assists the Bi-LSTM model in extracting the key features of dyslexia. In addition, rectified linear unit (ReLu) and Softmax layers were integrated for DD prediction. The early stopping strategies were used to monitor the validation loss and improve the model’s performance using the callback function to restore the best weights.

Fine-tuned LightGBM-based DDM

LightGBM is a gradient-boosting technique widely applied for classification and regression. A histogram-based algorithm was used to compute gradients. Compared to the existing gradient boosting technique, LightGBM produces a better result. It uses histogram-based algorithm for tree building, reducing the memory footprint. It supports out-of-core learning and distributed training that suits the proposed study to make predictions with minimum hardware configurations. In the context of DD, the class imbalance may cause challenges in identifying the optimal outcome. However, the inherent features, including weighted sampling and class-specific objectives, overcome the class imbalances. In addition, it can locate non-linear relationships between the features and the target variables. These advantages motivate the authors to employ the LightGBM model in this study.

Hyperband optimization was used to improve the performance of the LightGBM model. Using hyperparameter optimization, the authors fine-tuned the key parameters, including mm_leaves, max_depth, learning rate, n_estimator, reg_alpha, and reg_lambda. To control the step size during the boosting process, the authors set the ranges for learning rated from 0.01 to 0.1. An integer ranging from 3 to 10 was used for the maximum depth of each decision tree. The maximum number of trees per tree, ranging from 20 to 100, was used to optimize the LightGBM model. Feature fraction ranges from 0.5 to 1.0 were utilized for each split. The Hyperband algorithm evaluated the model using the randomly sampled hyper-parameter configuration. It iteratively evaluateed the performance and focused on resources associated with promising outcomes. The best-performing hyper-parameters configurations were selected based on the validation performance. Equation 4 shows the mathematical form of the fine-tuned LightGBM model. (4)${\displaystyle Prediction=hyperband\left(LightGBM\left(n,m,l,n,l1,l2\right)\right)}$

where n is the maximum number of leaves, m is the maximum depth of each tree, l is the learning rate, n is the number of boosting rounds, l1 is the L1 regularization, and l2 is the L2 regularization.

Evaluation metrics

To ensure the proposed model’s ability to capture the crucial features of dyslexia, the authors applied multiple evaluation metrics. Accuracy (Acc_y) was used to measure the percentage of similarity between predicted and actual values. Precision (Pre_c) refers to the accuracy of the model’s optimistic predictions. The higher value of precision indicates the absence of false positives. Recall (Rec_l) showed the percentage of true positives predicted by the proposed DD detection model. The existence of false negatives could be identified using the recall metric. F1-score integrated the outcomes of precision and recall. Cohen’s Kappa (K) was used to evaluate the model’s prediction in imbalanced datasets. In addition, the statistical significance of the model’s outcomes was computed using standard deviation (SD) and confidence interval (CI).