Automated sentiment analysis of visually impaired students’ audio feedback in virtual learning environments

Visual impairment and academic performance

Visual impairment, encompassing both total blindness and low vision, presents unique challenges in the academic context. These challenges necessitate adaptive teaching methodologies and curriculum designs to accommodate the specific learning needs of visually impaired students (Mwakyeja, 2013; Agesa, 2014). Notably, visually impaired learners have shown remarkable aptitude in language acquisition through auditory methods, often demonstrating enhanced verbal skills facilitated by active questioning and vocabulary enrichment (Ghafri, 2015).

The challenges of visual impairment in academic performance

Regarding the “Challenges of Visual Impairment in Academic Performance,” it is critical to note that much of the existing research on students with disabilities has been predominantly medical in focus, often overlooking their educational potential and specific challenges within higher education settings (Office of the Higher Education Commission, 2013). This study seeks to bridge this gap by introducing predictive models that leverage visually impaired students’ audio feedback to anticipate their learning outcomes (Elbourhamy, Najmi & Elfeky, 2023). This innovative approach aims to empower educators to fine-tune their instructional strategies, thereby more effectively addressing the unique learning needs of visually impaired students.

Student sentiments and learning

The results of the research on the relationship between sentiment and learning outcome are summarized in Table 1. The table presents results that have been found throughout studies conducted in various years. The studies indicated above give clear insight into the emotional dimension of learners’ involvement.

Sentiment analysis in education: a new approach for visually impaired students

However, sentiment analysis is traditionally applicable to domains that have such aims as dealing with unique educational needs, particularly in students with visual impairments. Some advanced methods of tracking sentiment are made possible by recent progress in natural language processing and machine learning, contrasting with the classic burdensome tests, which in general are impossible to apply feasibly for such students. This approach is based on sentiment classification, not only based on the polarity but also based on intensity through continuous and long-term monitoring for more targeted educational reactions. The following system architecture explains the proposed techniques.

Recently conducted meta-analyses and systematic reviews disclose that sentiment analysis tools can be effectively applied in educational contexts. The present study suggested that it could thus be a robust tool to be used not only in class but also in online and other multimedia settings to up students’ engagement level in learning (Xie & Luo, 2021; Nguyen et al., 2014; Gustafsson, 2020). To this effect, the study has spearheaded the ways of embedding the data-full-width = “true” sentiment analysis tools that would also be responsive to the visually impaired students—a field that has so far not been much explored. These tools help in decoding complex emotional signals, which are needed to generate feedback and support at par with the specific learning context of these students (Calvo & D’Mello, 2010).

In another study, the new framework incorporates categorical and dimensional approaches in the analysis of sentiment, particularly the valence-arousal space that captures emotional intensity and polarity in the application of sentiment analysis in multilinguistic settings (Kiritchenko & Mohammad, 2017; Bo & Lee, 2008). Such a framework has been noted, according to the need of the study, to enhance the challenges of arousal and valence detection that were identified in prior studies (Yu et al., 2016).

Based on the information provided

• 1. Specialized focus on educational sentiment analysis: This work pushes the frontier in terms of sentiment analysis in the domain of education, which is even less explored by prior works.

• 2. Improved sentiment tracking method on students: This is just an aspect in our system but with enhanced means of how the development of sentiments in students is tracked, mostly after the educational sessions. This is highly improved compared with other tools that analyze sentiments, but they do not have the sensitivity that is required to perfectly record the slight, small changes in sentiments occurring within the educational environment.

• 3. Nuanced sentimental classification: Going beyond the traditional three-tier (positive, neutral, negative) classification of sentiment in reviews, our article shall propose a more nuanced five-tier system (High Positive, Positive, Neutral, Negative, High Negative). This detailed categorization opens up deeper insights into the sentimental state of students which is very often missed in many other systems.

• 4. Filling a gap: Thus, by giving importance to the spontaneous feedback of one demographic group that had been otherwise neglected in previous analyses, the study fills a significant gap in the present research.

• 5. Focused on the visually impaired: This study would add value because it is focused on visually impaired students and is the first of its kind to deal with the sentimental responses of such a rare crop within an educational context.

• 6. Development of an innovative analytical system: We propose an advanced system that processes the audio feedback from visually impaired students to predict the performance of such students academically and sentimentally.

• 7. Educational data integration: The uniqueness of our system from most of the existing solutions lies in the fact that it integrates structured educational data with unstructured sentimental feedback, hence providing an overall insight into the learning experiences and the resulting outcome of the student. Such integrated data will further give greater accuracy and more practical conclusions for the stakeholders.

Data source

Our study involved 100 impaired vision students from various universities around Egypt and was carried out for 9 weeks. The course “Computer Skills Course in the English Language” was started through Microsoft Teams. All participants were communicated to, and forms of consent were signed before the course, whereby a choice of withdrawal/non-participation without penalties was given. The process of data collection was extensive and carefully integrated into the Microsoft Teams platform to ensure comprehensiveness and no loss of accuracy.

Throughout the course, multidimensional data were collected, including structured academic performance indicators and unstructured sentimental feedback. Both these kinds of data, when integrated, provided the opportunity to have an overall view of the progress and sentimental responses of the student throughout the course. Our data collection was multi-faceted and integrated into the Microsoft Teams platform. and we were gathering data as follows:

• Digital tracking on Microsoft Teams (structured data): We fully used the inbuilt functionalities of Microsoft Teams to track student participation. The same included checking attendance in each session, checking the submission rate of homework, and analyzing involvement in classroom discussions. In this regard, Microsoft Teams provided us with electronic logs and exported participation reports for analysis to quantify student engagement by using the “Insights” feature to get detailed reports on student engagement, which includes grades, attendance, and activity metrics. The first evaluation was done by the end of the fourth week, and the second one by the end of the ninth week.

• Collection of audio feedback (unstructured data): This work illustrates the study’s responsiveness, which is useful to help provide a useful means for identifying the needs of visual impairment in the section of the program where major assessments are needed. The first evaluation was done by the end of the fourth week, and the second one by the end of the ninth week, after audio feedback had been collected. The students were to audio-record themselves for every lesson while speaking, self-assessing, and reflecting on the learning content, then submit their recordings on the platform Microsoft Teams. The approach did not help in collecting valuable qualitative data but also provided in-depth insights into the experiences and understanding of the students.

In preparing the data for analysis, first, the audio feedback was normalized to have uniform levels of audio, hence making the inputs possess a uniform value. Then, the recordings went through voice activity detection technology for segmentation. Isolated spoken content needed to be obtained from the different interjections, background noises, and even silences to get only the spoken part of the data. This clean data input was then ready for transcription, setting the stage for further sentiment analysis and educational assessments.

The integration of these sources was structured and unstructured data, which brings the view of a student’s journey in academia into focus. This nuanced and varied approach allowed tracking academic performance in not just an objective manner but also gaining access to sentimental nuances, thus giving a larger perspective on the overall experience in the course for the students. For instance, the audio entries volume, which had been 2,648 in the fourth week, by the ninth week, was 4,237, indicative of an even active student participation level. These contained invaluable, rich audio feedback with insightful information in relation to the behaviors of students and how they were interacting with the material and instructors of the course.

Table 2 shows details of all structured and unstructured data sources, the types of data collected, from attendance records, homework completion, and class participation, to sentiments extracted from audio feedback, from high positive to high negative.

This methodological approach helps bring about a multifaceted understanding of the course’s impact on the student’s skills and sentimental responses.

Method for analysis

The study developed a flow chart to evaluate the effectiveness of an intelligence prediction system.

Figure 1 presents the tailored system developed for assessing visually impaired students’ sentimental responses via audio feedback. This sophisticated system is composed of three integral sub-systems: the Speech Recognition sub-system (STT), which transcribes audio to text; the Sentiment Analysis sub-system (SA), which evaluates the sentimental content of the feedback; and the Prediction sub-system, which anticipates students’ academic performance based on their sentimental feedback. Detailed descriptions of the functionality and interplay between these sub-systems follow in the subsequent section.

Ethical statement

The ethical committee in our Kafrelsheikh University has reviewed the study protocol and ethically approved the study under reference No. 334-38-44972-SD.

Sentiment expectation

In our study, the VADER lexicon-based approach was employed to categorize the sentiments within the freely given feedback from visually impaired students. This analysis occurred in both Week 4, with 2,648 feedback entries, and Week 9, with 4,237 entries. A noticeable increase in feedback by Week 9 indicated enhanced engagement among students. The sentiment categorization of this feedback is detailed in Table 3.

Table 3 presents a breakdown of emotional sentiment in feedback posts, categorized into five types: high negative, negative, natural, positive, and high positive. The data is organized into two sets, with the first column showing the number of feedback and their corresponding percentages out of total of 2,648 feedback, and the second column displaying the same for a total of 4,237 feedback. The table effectively illustrates the distribution of sentiment types among the feedback posts, highlighting the most and least frequent sentiment expressed.

To validate the accuracy of this lexicon-based approach, a panel of (3) experts specializing in student sentiment analysis was engaged. They analyzed a select sample of student comments, guided by a comprehensive manual specifically developed for this purpose (Appendix (1)). This manual contained detailed instructions to aid the experts in their analysis, categorizing sentiment into five distinct classes as per the prevailing research. This comparison was conducted on a select sample of 500 feedback entries Table 4 shows the comparison of sentiment categorization. The precision of the experts’ assessments was quantified using the following formula:

(4) $$accuracy={\displaystyle \frac{\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\mathrm{a}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{e}\mathrm{l}\mathrm{y}\mathrm{i}\mathrm{d}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{f}\mathrm{i}\mathrm{e}\mathrm{d}\mathrm{s}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{i}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{t}\mathrm{y}\mathrm{p}\mathrm{e}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{n}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{c}\mathrm{o}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{n}\mathrm{t}\mathrm{s}}}$$

Table 4 displays details of a Preliminary comparative analysis of the sentiment categorization by experts and the VADER system. This shows to what extent the machine categorization conforms to human judgment and it is highlighted by showing discrepancies in categorization. The feedback is divided into five types: High Positive, Positive, Neutral, Negative, and High Negative. The table shows how many feedback entries each expert and VADER placed in these categories. The table also shows the ‘Average Expert Count’ for each category, which is the average number the experts assigned to each sentiment type. Another important feature of the table is the ‘Discrepancy’ column, which shows the difference between the experts’ average count and VADER’s count for each sentiment.

The analysis of these discrepancies offers valuable insights into the reliability and alignment of automated sentiment analysis tools like VADER when compared with human judgment. This is especially pertinent in the context of feedback from visually impaired students, where understanding the sentimental landscape is crucial. The most and least frequent sentiments expressed, as identified by both the experts and VADER, are highlighted, providing a comprehensive understanding of the sentiments prevalent in the students’ feedback.

The comparison of VADER’s results with the ground truth established through human evaluation yielded an accuracy rate of 76%. This demonstrates a considerable level of reliability in VADER’s sentiment identification capabilities, affirming its suitability for analyzing feedback in educational settings involving visually impaired students.

The expectations of academic performance

The preliminary analysis leveraged a rich array of features, including advanced machine learning algorithms and both structured and unstructured data.

In our study, we utilized advanced machine learning algorithms to evaluate the potential for early prediction of student success at the 4th and 9th-week marks within a 12-week academic term. As outlined in Table 2 of our findings, we analyzed both structured data (such as student attendance records, participation in class, and homework completion rates) and unstructured data, the latter comprising sentimental tones derived from student remarks, as assessed by human annotators. We divided our dataset, consisting of information from 100 students, evenly into training and test groups, following a 50:50 split this decision was based on the unique characteristics of our dataset, which includes a relatively small sample size and a high level of diversity in the audio feedback from visually impaired students.

We acknowledge that while splits like 70:30 or 80:20 are more common in larger datasets, our decision for an even split was driven by the need to maximize the data available for testing, ensuring robust model validation. Given the novelty of our research area and the specific challenges posed by our dataset, a 50:50 split provided a more balanced approach to training and validation, allowing for a thorough assessment of the model’s performance on an equally representative test set.

Figure 2 displays our experimental results, demonstrating that our model was highly effective. By the fourth week, it had achieved an F-measure of 0.85 using SVM and 0.86 with CNN. By the ninth week, its performance was further improved, achieving an F-measure of 0.89 using SVM and 0.92 using CNN. The performance of the CNN-based approach was particularly impressive; it achieved the highest F-measure of 0.92, which underscores its superior effectiveness in our context.

Figure 3 compares the F-measures obtained by a CNN model with and without the inclusion of unstructured data to highlight the added value of sentiment analysis. The discernible improvement in F-measure when unstructured data is included indicates the effectiveness of utilizing self-rated feedback for early-stage predictions.

Table 5 shows accuracy of the SVM and CNN models over two periods of assessment is explained with these experimental results, while the table compares performance with that of structured and unstructured data to portray the impact of data types on prediction accuracy. thus, the empirical evidence suggests that incorporating sentimental feedback from visually impaired students markedly enhances the early predictive accuracy of our model. Such insights pave the way for educators to swiftly adapt their pedagogical approaches, optimizing educational support for visually impaired students in real-time.

Class-wise performance analysis of SVM and CNN

We have conducted an in-depth class-wise performance analysis of the SVM and CNN algorithms. This analysis is essential given the notable imbalance in our dataset, particularly the limited instances in classes such as high negative. So, we present a series of heatmaps Figs. 4A–4H, These figures provide the performance metrics of SVM and CNN under different data conditions, and assessment weeks, showing the models’ effectiveness in identifying the “at-risk” students by giving precision, recall, and F1-scores for “High Negative” class sentiment.

In our detailed class-wise performance analysis of the SVM and CNN algorithms, we address the challenges presented by the imbalance in our dataset, particularly the underrepresentation of crucial categories such as ‘high negative’. Figures 4A–4D illustrate the performance of the CNN algorithm during the 4th and 9th weeks, for structured data only and structured with unstructured data, and Figs. 4E–4H show similar analyses for the SVM algorithm.

The SVM achieved a precision of 0.8 for the ‘high negative’ class, suggesting that it was highly accurate in its predictions of at-risk students. Meanwhile, the recall for this class was 0.79 in the 9th week when analyzing combined structured and unstructured data, indicating that the SVM successfully identified a significant portion of the actual ‘high negative’ instances. This combination resulted in an F1-score of 0.85, reflecting a balanced harmonic mean of precision and recall, thus illustrating the model’s effectiveness at this stage.

Contrastingly, CNN’s performance in the 9th week when analyzing combined structured and unstructured data showed a precision of 0.74 for the ‘high negative’ class, a slight decrease compared to the SVM. However, the recall was 0.79, maintaining a strong ability to identify students at risk. The F1-score for this condition was 0.86, slightly higher than that of the SVM, suggesting that the CNN may be more adept at handling a combination of data types at this later stage.

These class-specific metrics allow us to draw direct correlations between the models’ predictive capabilities and their potential impact on the educational support for visually impaired students. The high recall rates are particularly important in educational settings were failing to identify at-risk students could result in missed opportunities for timely intervention. Conversely, the high precision rates are crucial for ensuring that resources are directed appropriately, avoiding the unnecessary allocation of interventions to students who do not require them.

Correlation analysis of teaching aspects and student sentiments

This subsection explores the relationships between various teaching aspects such as homework grades, clicks, attendance, and discussion participation, and their correlations with different student sentiment categories. Understanding these correlations is vital in assessing the impact of teaching methodologies on the sentimental and academic experiences of visually impaired students in virtual learning environments.

Figure 5 visually represents these correlations, using darker colors to indicate stronger correlations and lighter colors to show weaker ones.

According to Fig. 5, positive sentiments have a moderate positive correlation of 0.58 with structured data like attendance and homework grades. This correlation indicates that students who exhibit positive sentiments tend to have better academic outcomes, as reflected in higher attendance and improved homework performance.

Impact of Negative Sentiments. The analysis shows that negative sentiments have a weak correlation (0.1 or less) with academic metrics, suggesting that external factors significantly impact the academic performance of students with negative sentiments. The moderate correlation between positive sentiments and academic metrics emphasizes the importance of fostering a positive emotional environment that promotes engagement and boosts student success. While the weak correlation between negative sentiments and academic metrics highlights the need for additional non-academic support services to address the external factors affecting student performance.

Regression model for predicting student performance

This subsection presents a regression model summary, highlighting how various academic and engagement (structured data) factors collectively influence the performance of visually impaired students in a virtual learning setting Fig. 6 shows Regression model outputs showing predictors of student performance in virtual learning environments. This model quantifies the impact of several variables such as attendance and homework on students’ academic outcomes.

The regression analysis aimed to correlate the specific features of the sentiment analysis system, such as emotional granularity, with learning outcomes. The model’s correlation coefficient (R) of 0.507 indicates a moderate positive relationship between predicted and actual performance, showing that the features in the sentiment analysis system and other variables are useful in estimating student outcomes. Also, the R square value of 0.257 indicates that 25.7% of the variance in student performance can be explained by the variables used in the model, suggesting that these features provide valuable insights but that other unexplored factors also play a significant role.

Adjusted R square, at 0.226, provides a more accurate representation of the model’s ability to explain the variance in performance by accounting for the number of predictors. The Standard Error of the estimate, at 0.378, also shows the average deviation of observed values from the regression line, highlighting moderate prediction accuracy.

Together, these metrics offer a comprehensive overview of the factors influencing student performance in virtual learning environments, emphasizing the need for targeted strategies to enhance the educational experience for visually impaired students.

Information visualization

Figure 7 shows a screenshot from the login interface of the system, which requires users to enter their username and password to access their student profile. It was made very accessible by using speech and text, hence particularly helpful for students with visual impairments. The proposed system, depicted in Fig. 8, integrates structured and unstructured data to construct a multifaceted student profile. It features a dashboard offering a holistic view of academic assessments, tailored diagnostics, activity recommendations, and weekly sentimental states. The system employs auditory cues for conveying grades, social engagement, and class participation, enabling visually impaired students to receive personalized feedback. Notably, it employs sentiment analysis on self-assessment feedback, enhancing the early identification of at-risk students. This predictive capability is crucial for timely educational interventions and supports positive self-regulation learning practices. The system’s innovative use of sentimental valence graphs further aids students in reflecting on their learning journey, fostering an environment conducive to academic success.

The study’s main contributions are twofold. First, early identification of at-risk visually impaired students is crucial to the success of educational interventions aimed at changing the course of their academic performance. The proposed system provides diagnostic and suggested reports that offer useful information for positive sentiment self-regulation learning. The audio line graph of valence values also provides vital information for providing external guidance to students experiencing negative sentiments.