Improving course evaluation processes in higher education institutions: a modular system approach

Proposed system with SDRM approach

The overall structure of the system is presented in Fig. 1.

Construct a conceptual framework

In our study, which seeks to address how to improve the consistency and reliability of data in existing CIE processes and how to accelerate these processes, the functionalities and requirements of current systems were examined in detail. Upon closer inspection, it was observed that inconsistencies arise between the responses students provide to structured questions and the opinions they write in the open-ended section in the CIE process, forming a systematic issue. Additionally, it was found that examining the data obtained from the CIE leads to a significant loss of time for administrators.

While developing the conceptual framework to guide the creation of a system intended to propose solutions to these problems, the following considerations were taken into account:

Develop a system architecture

At this stage, the goal is to design the overall structure of the system to achieve the objectives of the research. The system architecture includes the necessary modules for accurately and consistently processing the data collected from students, as well as the interactions among these modules.

The developed architecture combines core functionalities in a modular structure. The proposed system comprises five modules: data collection, data preprocessing, sentiment analysis, inconsistency detection, and reporting (Fig. 2). Adopting a modular structure is crucial, as each higher education institution may have different CIE processes, and it is therefore important to adapt the system to suit each institution’s course evaluation practices.

Designing the system

At this stage, the goal is to develop a design that enables the efficient execution of all modules, including data collection, processing, sentiment analysis, inconsistency detection, and reporting. This process involves the planned integration of supervised machine learning algorithms, data preprocessing techniques, and consistency control mechanisms to process the CIE data gathered from students.

Data collection module design

In this study, the data collection module is based on CIEs obtained online from the student information system of a foundation university (referred to as University A) in Turkey, which has 18,000 (17,908) students. This model can be easily adapted to other higher education institutions that use different student information systems.

At this stage, in order to train the machine learning algorithms to be used in the other modules of the system, all CIEs for the courses offered in the last five years at the aforementioned university were collected via the student information system. The collected data comprises 16 structured questions (scored on a scale of 1 to 4) and one open-ended question for each CIE. For students, the first 16 questions in the CIE are mandatory, whereas the open-ended question is optional. Since the goal of this study is to detect inconsistencies between the sentiment of the open-ended question response and that of the structured questions, CIEs without responses to the open-ended question were excluded from the dataset. However, if this system is put into practice, the CIEs without an open-ended response should be included, based on the sentiment derived from the structured questions.

Under these conditions, 13.651 CIEs corresponding to 2.874 different courses meeting the specified criteria at University A were identified. The data were stripped of all identifying information and anonymized before being provided to us (University A Ethics Committee Report Document No: 17162298.600-11).

The internal consistency of the instrument was assessed using Cronbach’s Alpha coefficient, which yielded a value of 0.992 across 16 items. According to widely accepted benchmarks, a Cronbach’s Alpha value above 0.90 indicates excellent reliability. Therefore, the obtained coefficient suggests that the items within the survey demonstrate a very high level of internal consistency and measure the intended construct reliably. This level of reliability supports the use of the instrument for further statistical analyses and strengthens the validity of the findings derived from the data (Table 1).

Data preprocessing module

Data preprocessing is a crucial step that involves converting raw data into a format suitable for analysis. In this step, both types of data in the dataset (text responses to open-ended questions and quantitative responses to structured questions) are labeled, and any elements that do not affect the sentiment are removed.

Operations carried out in the data preprocessing module (Fig. 3):

1. Text data labeling: The 13,651 CIE records obtained from the data collection module were manually labeled by experts as positive (1), negative (−1), or neutral (0). As a result of this process, 2,914 neutral, 1,955 negative, and 8,782 positive comments were identified and labeled in the dataset.

   This labeling step will be performed automatically by the chosen algorithm in the sentiment analysis module once the system is implemented. It was done manually here with the aim of determining the most reliable algorithm for sentiment analysis (to be used during the training stage of the algorithm).

2. Numeric data labeling: In the CIE data, 16 structured questions were rated by students on a scale of 1 to 4 (1—Not satisfied at all, 2—Not satisfied, 3—Satisfied, 4—Very satisfied). In the resulting dataset, an arithmetic mean of the 16 questions was calculated. If this mean (i) was 3 or above, it was labeled as positive (1); (ii) between 2 and 3, neutral (0); and (iii) below 2, negative (−1), indicating the student’s overall sentiment for that specific CIE.

3. Text data cleaning: During the data cleaning phase, uppercase letters, punctuation marks, numbers, emojis, and stop words (such as “and,” “or,” “etc.”)—which do not affect the sentiment within a sentence—were removed from the dataset.

4. Text data vectorization: The texts were vectorized using the TF-IDF method. TF-IDF is used to determine the importance of a word in a document; frequently occurring words (high term frequency) are considered important within that document. However, if such words also appear commonly in other documents (low inverse document frequency), their importance is reduced, emphasizing the uniqueness of the word. An example of TF-IDF is illustrated in Table 2.

Sentiment analysis module

Sentiment analysis is a field concerned with the computational processing of opinions, emotions, and subjectivity in texts, aiming to develop sentiment-focused information retrieval systems (Pang & Lee, 2008). The main objective of this process is to objectively evaluate the emotional expressions in students’ open-ended responses. In this module, the machine learning algorithm to be used for sentiment analysis in the system has been selected. Therefore, the preprocessed texts are transferred to supervised machine learning algorithms (support vector machines (SVM), naive Bayes, random forest, decision trees, K-nearest neighbors (KNN), and GPT-4-Turbo Preview), and each text is labeled as positive, negative, or neutral by the respective algorithm.

SVM attempts to find a hyperplane that separates classes by transforming the data into a high-dimensional space. This hyperplane maximizes the minimum distance to the data points closest to the boundary between classes. For non-linearly separable cases, kernel functions are used to project the data into higher dimensions (Cortes & Vapnik, 1995).

Naive Bayes operates based on Bayes’ theorem and assumes that all features are independent of one another. The probability of belonging to a class is calculated by multiplying the conditional probabilities of each feature. It is commonly used in applications such as text classification and spam filtering (Maron, 1961).

Random forest combines multiple decision trees through an ensemble learning method. Each tree is trained on a random subset of the data, and predictions are combined through a voting mechanism. This approach reduces overfitting and ensures high accuracy (Breiman, 2001).

Decision trees carry out classification or regression by recursively splitting the data. At each node, the data is divided into two subgroups based on a feature and a threshold value. The final leaves represent the predicted classes or values. The algorithm selects the best split according to criteria like information gain or entropy (Quinlan, 1986).

KNN measures the distance of a data point to its K-nearest neighbors for classification or regression. A new data point is classified based on the majority of these neighbors. KNN is a non-parametric method that makes comparisons directly with the data rather than undergoing a training phase (Cover & Hart, 1967).

GPT-4 Turbo Preview is a large language model based on OpenAI’s Transformer architecture, designed to provide a faster and more cost-effective solution for natural language processing tasks. The model first undergoes pre-training on a broad collection of text to learn linguistic structures, followed by fine-tuning to optimize for user-specific tasks. “Turbo” optimizations offer lower computational costs and faster response times compared to GPT-4 (OpenAI, 2023).

Supervised machine learning algorithms are preferred for sentiment analysis in Turkish texts due to their ability to learn from labeled datasets and accurately classify texts in context (Tokcaer, 2021; Akgül, Ertano & Banu, 2016; Tuzcu, 2020; İlhan & Sağaltıcı, 2020).

Using the sklearn library, data split into an 80:20 ratio was employed to train the machine learning algorithms mentioned above, targeting high accuracy. It is a common practice to allocate 80% of the dataset for training and 20% for testing when evaluating the performance of machine learning models (Zilyas & Yılmaz, 2023; Almosawi & Mahmood, 2022; Ramteke et al., 2016; Putra et al., 2022). During training, the algorithms parameters were kept at default values, and accuracy rates were examined after training.

Accuracy

Accuracy is a metric that measures how correct a model’s predictions are across the entire dataset. It represents the ratio of correctly classified examples to the total number of examples in the dataset and takes into account both true positive (TP) and true negative (TN) predictions.

$$Accuracy={\displaystyle \frac{NumberofCorrectPredictions}{TotalNumberofData}}$$

Mean average precision

Precision is the ratio of truly positive examples among those predicted as positive by the model. Macro average precision calculates precision values for each class separately and then takes the arithmetic mean of those values.

$$Precision={\displaystyle \frac{TruePositive}{TruePositive+FalsePositive}}$$

$$MacroAvgPrecision={\displaystyle \frac{1}{N}\sum _{i=1}^{N}Precision\left(i\right)}$$

Macro average recall

Recall measures how accurately the model identifies truly positive examples as positive. Macro average recall calculates recall values for each class separately and then computes their average.

$$Recall={\displaystyle \frac{TruePositive}{TruePositive+FalseNegative}}$$

$$MacroAvgRecall={\displaystyle \frac{1}{N}\sum _{i=1}^{N}Recall\left(i\right)}$$

Table 3 presents a comparison of the sentiment analysis performance of the algorithms in terms of accuracy, precision, and recall. The results show that the “GPT-4-turbo-preview” model provided by OpenAI achieved the highest performance across all criteria. With 85% accuracy, 88% precision, and 95% recall, this model outperformed the other algorithms. Hence, the GPT-4-Turbo preview algorithm was chosen to be used in the proposed system. The performance comparison across different algorithms reveals that GPT-4 Turbo preview outperforms all traditional machine learning models in terms of accuracy (85%), mean average precision (88%), and mean average recall (95%). Among the classical algorithms, SVM demonstrate the most balanced performance, achieving relatively high scores across all metrics (83% accuracy, 76% precision, and 76% recall). Naive Bayes and random forest follow closely, though with slightly lower overall effectiveness. Decision trees and KNN show the weakest performance, particularly in recall, where KNN achieves only 58%. These results suggest that while traditional models offer competitive baselines, large language models like GPT-4 provide a notable advantage, particularly in capturing nuanced patterns relevant to precision and recall.

Inconsistency detection module

This module examines the consistency of each CIE labeled in previous steps. Potential inconsistencies are identified by comparing the sentiment label of the textual data with that of the corresponding numeric data (Table 4). For instance, when a student provides a low numeric score but expresses positive sentiments in the open-ended section (or vice versa), this situation is deemed inconsistent.

Final report module

The final report consolidates the results of all analysis and detection processes to present an overall assessment of course and instructor performance. Its main goal is to provide concrete and reliable information to enhance the accuracy and transparency of course evaluation processes in higher education institutions. The report includes positive, negative, and neutral sentiment trends that summarize student perceptions of the course or instructor. Moreover, by visualizing and evaluating the results obtained after removing inconsistent data, it contributes to strategic decision-making processes for administrators and instructors.

Depending on the institution’s needs or the authorization level of the individual viewing the report (administrators, course instructors, etc.), the specific information included in the report may vary. However, the final report designed for a course in the proposed system contains the following information (Attachment 1):

• Pie chart displaying the sentiment distribution of consistent feedback on the course

• The average numeric scores for the course

• The highest and lowest average scores given in evaluations

• The percentage of all comments that are consistent/inconsistent

Build the (prototype) system

At this stage, in order to test the applicability of the developed model in a real course evaluation scenario, a prototype system was created. The prototype system was applied to the CIEs of a course containing 431 comments and survey data. This process is illustrated in Fig. 4.

In the prototype system, the GPT-4-Turbo Preview model, which achieved the highest accuracy rate for sentiment analysis, was used. No changes were made to the operations carried out by the system’s other modules.

The dataset employed in this study comprises 431 CIE responses collected within the past academic year from a single course. The sentiment annotation of the qualitative feedback was performed manually by a domain expert with expertise in natural language processing and educational technologies. The majority of the inconsistencies stem from cases where positive quantitative evaluations are accompanied by negative qualitative comments.

As a result of the analyses, out of 431 CIEs, 161 were deemed inconsistent and 270 were deemed consistent. This finding indicates that 37% of the collected CIEs contain misleading results.

A final report was generated using the 270 CIEs labeled as consistent. The following elements are planned to be included in this report.

Providing administrators and instructors with meaningful insights into the strengths and weaknesses of the course, thus helping them make more informed decisions in future course planning and educational processes.

Observe and evaluate the system

The prototype system was tested on 431 comments and survey data collected for a single course. In this prototype, the sentiment analysis conducted by OpenAI achieved an accuracy rate of 86%.

Upon examining the sentiment labeling results, it was observed that OpenAI generally had difficulty distinguishing whether a comment was positive or neutral. While it performed well in correctly identifying both positive and negative comments, its success rate decreased when differentiating between positive and neutral comments. As seen in the prototype system, removing inconsistent data has led to a final report grounded in more reliable results.