Automatic detection of teacher behavior in classroom videos using AlphaPose and Faster R-CNN algorithms

Process introduction

The experimental process is shown in Fig. 1.

AlphaPose is an efficient open-source human pose estimation system capable of detecting and tracking multiple persons’ body joints in real-time. In this study, each frame in the PCB dataset is processed through AlphaPose to extract 17 key skeletal points per person, including joint coordinates of the head, shoulders, elbows, wrists, hips, knees, and ankles. These key points are then converted into feature vectors representing spatial relationships of limb positions. To ensure scale invariance and consistency across frames, all coordinate values are normalized by image width and height.

Prior to feeding the data into the Faster R-CNN classifier, we perform data augmentation techniques, including horizontal flipping and Gaussian noise injection, to enhance model robustness. The processed keypoint vectors form a high-dimensional feature representation that encapsulates the teacher’s motion and posture information, which is then used by the classifier to identify inappropriate behavior patterns. This method can be used for the analysis of classroom videos and the detection of teacher behavior. Compared to single behavioral classification methods, combining AlphaPose with Faster R-CNN classification methods results in higher recognition accuracy. The process of behavioral classification is shown in Fig. 2.

Dataset introduction

We used a PCB dataset to train a model that classifies the behavior of teachers. The dataset includes 74 video clips with a total of 51,800 labeled frames after processing. The data set is available at https://doi.org/10.1007/s00521-020-05587-y. Each frame contains multiple preschoolers and one teacher and is labeled either 0 (no inappropriate behavior by the teacher towards the children) or 1 (inappropriate behavior by the teacher towards the children). A total of 2,937 marked images were identified as containing improper actions, which represents 5.67% of the overall number of frames. To optimize our data usage, we partitioned the dataset into training and test sets, maintaining a ratio of 7:3. The distribution of data types after partitioning is presented in Table 1.

To improve the generalization ability of the model and simulate the diversity of real-world classroom environments, a series of data augmentation techniques were applied to the training set. These include:

• Random horizontal flipping: to account for mirrored teacher gestures and body orientations.

• Random rotation (±15 degrees): to simulate camera angle variations and minor shifts during video capture.

• Random scaling and cropping: to emulate varying distances between the camera and subjects, as well as partial occlusions.

• Brightness and contrast adjustments: to reflect lighting changes that may occur across different classroom conditions.

These augmentation strategies were implemented using standard OpenCV and PyTorch-based data pipelines before feature extraction by the AlphaPose algorithm.

Experimental method

After the preparation work on the dataset is completed, we use the AlphaPose model to recognize the body limbs of people in the video dataset. AlphaPose is an efficient open-source human pose estimation system that can achieve multiple-person pose estimation, pose tracking and other functions.

The video frames recognized by AlphaPose will be labeled with the body information of the children and teachers. Compared to the original video, data that has been labeled is easier to classify correctly.

In order to avoid the loss function failing to converge during training and enhance the generalization of the model, it is necessary to normalize the sample data (Liu et al., 2025; Wang et al., 2022). This article mainly performs normalization processing on the skeleton point coordinates extracted by the AlphaPose model, as shown in Formula (1).

(1) $$\left(\dot{x},\dot{y}\right)=\left({\displaystyle \frac{x}{w},{\displaystyle \frac{y}{h}}}\right).$$

Among them, $x$ and $y$ are the $x$ coordinate and $y$ coordinate of the skeletal points in the image, respectively. $w$ denotes the width of the image, and $h$ denotes the height of the image. Ultimately, the normalized coordinates $\left(\dot{x},\dot{y}\right)$ are obtained.

We divided the training set and test set in a ratio of 7:3 during the training process. We chose the Sigmoid function as the activation function, which is represented as Formula (2). In this article, recognizing teacher behavior is a binary classification problem, and we selected the binary cross-entropy loss function as our loss function. Its specific calculation method is shown in Formula (3).

(2) $$sigmoid\left(y\right)={\displaystyle \frac{1}{1+{\mathrm{e}}^{-y}}}$$

(3) $$L\left(y,\hat{y}\right)=-{\displaystyle \frac{1}{2}\left[\hat{y}\mathrm{l}\mathrm{o}\mathrm{g}\left(sigmoid\left(y\right)\right)\right]+\left(1-\hat{y}\right)\mathrm{l}\mathrm{o}\mathrm{g}\left(sigmoid\left(1-y\right)\right).}$$

Among them, $y$ represents the actual neuron, and $\hat{y}$ represents the true label marked in the dataset. 0 indicates normal behavior, while 1 indicates abnormal behavior.

Compared to other convolutional neural network models, Faster R-CNN performs well in terms of both detection speed and classification accuracy. Therefore, this article uses this network to classify images processed by AlphaPose.

Faster R-CNN, a deep learning-based object detection algorithm by Ross Girshick in 2015 and updated in 2016, represents an improved version of R-CNN and Fast R-CNN. It introduces the region proposal network (RPN) to replace the Selective Search algorithm for object extraction, thereby enhancing both detection speed and accuracy.

The Faster R-CNN model consists of two principal components: the region extraction network and the object classification network. The RPN, a sub-network based on convolutional neural networks, generates candidate object regions in the image. RPN slides a small window on the feature map, and each window outputs the scores of k (usually 9) anchor boxes and their corresponding bounding box regression parameters. These anchor boxes have different ratios and scales, covering objects of varying sizes and shapes. Then, based on the scores of these anchor boxes, the non-maximum suppression (NMS) algorithm selects the most likely candidate regions containing the object, which are then sent to the object classification network. The object classification network classifies and locates the regions. It receives the candidate regions extracted from the RPN and classifies and locates each region. This network usually uses a fully convolutional network, which can process regions of different sizes and output which category each region belongs to and its corresponding bounding box position.

The Faster R-CNN training process consists of two steps: first, the RPN network generates candidate regions and classifies and locates them. Then, the network is trained using the cross-entropy loss function and the smooth L1 loss function to correctly classify and locate the object regions, as shown in Formulas (4) and (5).

(4) $$Loss=-\sum _{i=0}^n{x}_{i}log\left({\hat{x}}_i\right)$$

(5) $$Smooth{L}_1=\{\begin{array}{c}\left|x\right|-0.5\left|x\right|>1\\ 0.5x^2\left|x\right|<1.\end{array}$$

Relative to Fast R-CNN, Faster R-CNN has made significant strides in detection speed and accuracy. Its speed is more than ten times faster than Fast R-CNN, and its accuracy has also improved. Innovative in region extraction and object classification, Faster R-CNN stands as one of the most advanced object detection algorithms to date.

Evaluation method

After completing the training, we can obtain a Faster R-CNN model for the behavior classification of teachers. For each image, the output of the classification model consists of two classes: 0 represents normal teacher behavior and 1 represents inappropriate teacher behavior. The classification results include four categories: true positive (TP) indicates that positive samples are correctly classified as positive by the model; false positive (FP) indicates that negative samples are incorrectly classified as positive by the model; false negative (FN) indicates that positive samples are incorrectly classified as negative by the model; and true negative (TN) indicates that negative samples are correctly classified as negative by the model. Evaluating algorithm performance requires considering both precision (P) and recall (R), with formulas for calculating precision and recall given in Formulas (6) and (7), respectively (Torgo & Ribeiro, 2009). Since recognition models in this scenario tend to favor recall rate, we selected F2-score, a commonly used comprehensive evaluation index for classification models, as our evaluation standard using Formula (8).

(6) $$Precision={\displaystyle \frac{TP}{TP+FP}}$$

(7) $$Recall={\displaystyle \frac{TP}{TP+FN}}$$

(8) $$F_2\text{-}score={\displaystyle \frac{5\ast Recall\ast Precision}{4\ast Recall+Precision}.}$$

Dataset preview

Using the method introduced in the previous section, this article uses a PCB dataset. First, the AlphaPose algorithm is used to recognize actions in all labeled images in the dataset. The recognition results of the dataset include limb information of undergraduates and teachers in the images. The position information of human limbs is essentially the position information of limb key nodes, which can be regarded as multidimensional vectors. To enhance the visual representation of the data, we applied the t-SNE algorithm to project the data onto a two-dimensional plane for display, as depicted in Fig. 3.

As evident from the graph, a majority of the data mapping points are situated in the central region of the chart, whereas a negligible proportion of the data points are dispersed near the central cluster, signifying anomalous characteristics. The aberrant features of these data points are indicative of teacher misconduct. The t-SNE projection reveals a distinguishable clustering pattern, where outlier points corresponding to inappropriate teacher behaviors are separable from the central cluster. This demonstrates the discriminative capability of AlphaPose features.

Model training

During the training phase, 70% of the available data was utilized as the training set. The training set consisted of 34,204 frames representing proper behavior and 2,056 frames displaying improper behavior, resulting in a total of 36,260 frames. Figure 4 illustrates the change in loss function as training progressed.

The figure depicts a gradual decrease in the loss function value as the number of training rounds increases. Initially, the loss function value rapidly diminishes with the increase of training rounds, followed by a conspicuous jolt at around 14 rounds. Subsequently, the value continues to decrease quickly. However, at approximately 35 rounds, the loss function value exhibits a minor amplitude oscillation, and it no longer decreases below 0.3, signifying that the model’s performance has reached a bottleneck. After around 70 rounds, the model was optimized and escaped from the local optimal solution. The loss function value no longer reaches 0.4 and remains minimal. As training rounds progress, the loss function value continues to gradually decline. The loss function curve shows stable convergence after approximately 100 epochs, indicating that the model achieves a satisfactory trade-off between training accuracy and generalization.

Evaluation and results

After training is complete, the model is tested using test set data and the results are shown in the confusion matrix in Fig. 5.

The problem addressed in this article is a binary classification problem. Thus, the confusion matrix is a 2 * 2 matrix. As can be seen from the figure, TN is 42831 and TP is 6672. The precision rate is 0.7489, and the recall rate is 0.9911.

In this particular scenario, the recognition model prioritizes recall rate, and the evaluation standard for the model in this article is the F2-score. Table 1 presents a comparison between the impact of the Faster R-CNN classification method and other models following AlphaPose processing.

To quantify the contribution of AlphaPose in our framework, we interpret the Faster R-CNN model in Table 2 as a baseline without AlphaPose preprocessing. Compared to this configuration, the full system incorporating AlphaPose achieved a 1.03% improvement in F2-score and a 1.04% improvement in recall. This clearly demonstrates that the pose-based features extracted by AlphaPose significantly enhance classification accuracy by encoding motion and spatial cues that are otherwise difficult to capture using raw image features alone. The results presented in the table unequivocally demonstrate the superiority of the method posited in this article over other models. The proposed method boasts an accuracy rate that surpasses that of the basic Faster R-CNN model by 1.3% and a recall rate that surpasses the same model by 1.06%. In comparison to traditional CNN and SVM, the proposed method achieves an improvement in recall rate by 10.1% and 7.4%, respectively. Furthermore, the F2-score of the proposed method outperforms all other models, with a score that surpasses that of CNN, SVM and Faster R-CNN by 6.09%, 3.67% and 1.19%, respectively. This improvement highlights the advantage of integrating pose estimation with object detection for behavioral classification. The combination of AlphaPose and Faster R-CNN significantly enhances the detection accuracy for inappropriate teacher behaviors, which is critical for practical deployment in educational monitoring systems.

In a practical application in classrooms, the accuracy rate of the AlphaPose + Faster R-CNN classification method can reach 74.89%. As shown in the Fig. 6, after using this method, education staff reduced their time for recording and analyzing each child’s behavior by 47%. There was a reduction of 63% in incidents of inappropriate behavior by teachers.

Experiment summary

Firstly, the PCB dataset is used as training data in this article. Then, the AlphaPose algorithm is used to identify the position of teachers’ limbs, and the video with limb position information is input into the Faster R-CNN classification network for behavior classification. Experimental results show that the AlphaPose+Faster R-CNN model performs better than several other models. In the test set, TN is 42831, TP is 6672, the precision rate is 0.7489 and the recall rate is 0.9911. In actual classroom application, this system’s accuracy can reach 74.89%, reducing educators’ time spent on recording and analyzing each teacher’s behavior by 47% while decreasing inappropriate behaviors by teachers by 63%. Finally, this article analyzes and discusses the applications and prospects of Faster R-CNN classification methods based on local posture in teacher behavior that is under increasing attention.

Contribution

Moreover, this study underscores the immense potential of utilizing cutting-edge computer vision and machine learning techniques in the sphere of education. As technology continues to advance, there is an escalating necessity to incorporate these techniques into educational settings to augment teaching and learning outcomes. For instance, computer vision algorithms can be employed to monitor student engagement and attention levels during classroom activities, which can enable teachers to modify their teaching strategies and provide individualized feedback to students. Additionally, machine learning models can be utilized to scrutinize student performance data and offer personalized learning recommendations based on their unique strengths and weaknesses.

Prospect

Another promising avenue of research in the realm of computers and education is the creation of intelligent tutoring systems. These systems can use artificial intelligence algorithms to deliver personalized learning experiences for students based on their specific learning styles and requirements. By analyzing student data in real time, these systems can adapt their instructional techniques and offer targeted feedback to facilitate students in accomplishing their learning objectives.

In summary, while this study constitutes a substantial advancement in the field of computers and education, there is still a considerable amount of work to be done to realize the full potential of technology in enhancing teaching and learning outcomes. Future research can concentrate on developing more sophisticated models and analysis frameworks, incorporating multimodal data, and exploring new applications of computer vision and machine learning in educational settings. Ultimately, the objective is to generate more effective and personalized learning experiences for students, thereby enhancing overall educational outcomes.

Deficiency

It is worth noting that the dataset used in this article is limited in scope and sample size. Future studies may consider using larger datasets to verify the results of this study. In addition, to further improve behavior recognition accuracy, future research can more deeply optimize the AlphaPose algorithm and Faster R-CNN classification network to achieve better classification performance in broader contexts.