Filipino sign language alphabet recognition using Persistent Homology Classification algorithm

Persistent homology

Persistent homology (PH) is a method widely used in topological data analysis (TDA). It can be used for determining invariant features or topological properties of a space of points that persist across multiple resolutions (Carlsson, 2009; Edelsbrunner & Harer, 2008). These invariant features capture the qualitative properties of data due to their sensitivity to small changes in the input parameters, making PH favored by researchers. PH application extends to different data types such as point clouds, images, time series, etc. In this article, we focus on the computation of PH on point clouds.

A point cloud $(X,d)$ represents a finite set of points X together with a distance function $d$. The usual assumption is that X is sampled from an underlying topological space $\mathbb{S}$. However, describing the topology of $\mathbb{S}$ based on sample points of X is not easy. This is where PH can be used.

In computing PH, a filtration is constructed using X, converting the point cloud into a nested sequence of simplicial complexes. A visual description of this process is presented in Fig. 1. This is done by defining a non-negative real number $ϵ$ that serves as a parameter to thicken X. We denote the thickened point cloud corresponding to the parameter $ϵ$ as $X_{ϵ}$. As the value of $ϵ$ increases, simplices are added to the complexes, and a sequence of nested simplicial complexes is formed.

Adding new simplices to the complexes can be done using a variety of ways. In this article, Vietoris-Rips (VR) complex is used. In VR complex, two points $x_i$ and $x_j$, each of which are initially 0-simplex, are connected when the distance $d(x_i,x_j)\le 2ϵ$. This forms a 1-simplex or a line segment. Adding another point $x_k$ that satisfies $d(x_i,x_k)\le 2ϵ$ and $d(x_j,x_k)\le 2ϵ$ forms a 2-simplex or a triangle. Adding another point $x_l$ that satisfies $d(x_a,x_l)\le 2ϵ$ for $a=i,j,k$ forms a 3-simplex or a tetrahedron, and so on. It is worth noting that the parameter $ϵ$ is the only parameter changing in this filtration process. The addition of simplices depends on this $ϵ$ value and the distances of 0-simplices from one another.

In each filtration step, the homology groups are extracted. These are invariant features of a topological space that provide important information and can be computed algebraically. The homology of the underlying topological space $\mathbb{S}$ can be approximated by the homology of the simplicial complexes derived from X.

For this, suppose K is a finite simplicial complex and $K_1\subseteq K_2\subseteq \dots \subseteq K_r=K$ is a finite sequence of nested subcomplexes of K. Here, K is called a filtered simplicial complex and the sequence $\{K_1,K_2,\dots \}$ is the filtration of K. The homology of each of the subcomplex can be computed as follows. For each $k$, the inclusion maps $K_i\to K_j$ induce ${\mathbb{F}}_2$-linear maps $f_i^j:H_k(K_i)\to H_k(K_j)$ for all $i,j\in 1,2,\dots ,r$ with $i\le j$. It follows from functoriality that ${f_l}^j\circ f_i^l=f_i^j$ for all $i\le l\le j$.

Now suppose $K_s$ is a subcomplex in the filtration or a filtered complex at time $s$. We define the $k$-th cycle group of $K_s$ as $Z_k^s=Ker{\mathrm{\partial }}_k^s$ and the boundary group of $K_s$ as $B_k^s=Im{\mathrm{\partial }}_{k+1}^s$. Then, the $k$-th homology group of $K_s$ is given by

(1)

Consequently, for $p\in \{0,1,2,\dots \}$, the $p$-th persistent $k$-th homology group of K given a subcomplex $K_s$ is

(2)

Simply, for each nonnegative integer $k$, there exists a $k$-th homology group $H_k(X_{ϵ})$ representing $X_{ϵ}$. The 0-th dimensional, 1-dimensional, and 2-dimensional homology groups gives the connected components, holes or tunnels, and voids, respectively. These algebraic structures are homotopy invariant, meaning they do not change when the space undergoes bending, stretching, or other deformations, making them ideal as representation of data.

Persistence diagram and barcode

The result of obtaining the homology of the filtered complexes can be represented using a persistence diagram or persistence barcode. Example of some filtration process and their corresponding diagrams and barcodes are shown in Fig. 2. These representations show the appearance (birth) and disappearance (death) of intrinsic topological features, such as homology groups and Betti numbers. In other words, these birth and death values represent the filtration index (or the parameter $ϵ$) at which the topological feature appear or disappear, respectively. The lifespan or duration of these topological properties are essential for the qualitative analysis of the topology of the data. Shorter lifespan are often associated with noise while longer ones are the important topological features. This lifespan parameter will be essential for the development of the topology-based classifier PHCA. For a more comprehensive discussion of the computation of PH, the reader is referred to Edelsbrunner & Harer (2008).

Alternatively, the persistence diagram and barcode obtained from the point cloud X can also be represented as an $n\times 3$ matrix, denoted as the persistence $\mathcal{P}(X)$. The number of rows $n$ represents the number of topological features or the total number of 0-dimensional holes, 1-dimensional holes, 2-dimensional holes, and so on, depending on the defined maximum dimensions that can be detected during the filtration. Meanwhile, the first, second, and third column entries of the $q$-th row of $\mathcal{P}(X)$ represents the dimension, birth, and death times of the $q$-th topological feature in the filtration of X, respectively.

Now, since that the filtration process cannot be performed in an infinite duration of time, a maximum scale $maxsc$ must be defined. Scale in this context represents the $ϵ$ value or the distance threshold. In practice,

(3)

Training and classification using PHCA

Suppose X is the training dataset consisting of $m$-dimensional data points categorized into $k$ distinct classes. More specifically, suppose that $X=X_1\cup X_2\cup \dots \cup X_k$ where each $X_i$ is the set of data points in class $i$ for $i=1,2,\dots ,k$. We note that $X_i\cap X_j$ for $i\ne j$ implying that no two classes contains the same data point. Introduced with a new data point $\alpha$, we want to determine which class does this point belong to.

The training process of PHCA involves computing for the persistence of each class, $\mathcal{P}(X_i)$ for $i=1,2,\dots ,k$. Then, the model measures the topological effect of introducing $\alpha$ to each of these classes. For this, the model defines $Y_i=X_i\cup \{\alpha \}$ and compute for $\mathcal{P}(Y_i)$ for $i=1,2,\dots ,k$. This process records the changes in PH between $X_i$ and $Y_i$ for $i=1,2,\dots ,k$. The new data point is classified to the class which results in the minimum change in PH. This change is measured using the score function discussed in the next section.

Score function for PHCA

After training the PHCA model, scoring each of the classes is necessary to choose which class does the new data point $\alpha$ belongs. For this, the model computes for $Score(X_i)$ for $i=1,2,\dots ,k$ and compare their results. Recall that the PH of a point cloud can be represented as an $n\times 3$ matrix where $n$ represents the number of topological features and the three columns represent the dimension, birth, and death of each topological feature, respectively. From here, we define the lifespan of the $q$-th topological feature as $l_q=d_q-b_q$ where $b_q$ and $d_q$ are the birth and death times of the $q$-th topological feature, respectively. Then, we can define the score function as

(4)

(5)

Data description

The dataset used in this article is published by Porton (2023) in Kaggle. It consists of 11,700 images of hands that corresponds to FSL signs of a letter in the alphabet. This article only focuses on the static signs. Hence, we omit the images from the letters J and Z.

Each of the 10,800 images considered in this article has dimension 300 pixels $\times$ 300 pixels $\times$ 3 channels (RGB). There are 450 images for each class or letter of the alphabet, except for J and Z. The images are obtained using a camera that captures a video feed. For each class, the signer performs a sign and a video is captured. Then, frames from the captured feed are extracted as images which forms this dataset. Various lighting and background conditions can be observed in the images, increasing variability in classification.

Classification scheme

We discuss here the classification scheme implemented in this article. Figure 3 shows the summary of this classification scheme. The main stages are Feature Extraction, Data Splitting, Feature Scaling, Hyperparameter Tuning, and Classification. This scheme is implemented for each trial, totaling up to 10 trials in this article.

Feature extraction

The feature extraction process mainly utilized the MediaPipe Hands pipeline (Zhang et al., 2020) illustrated in Fig. 3A. The structure of the pipeline involves a palm detection model and a hand landmarker model. The palm detection model reduces the complexity of the task by first estimating a bounding box on the palm. Precise keypoint localization is then implemented using the hand landmarker model to extract the 21 three-dimensional hand-knuckle coordinates. Each landmark consists of $x$ and $y$ coordinates and the $z$-value represents the depth with respect to the camera. Figure 4 presents all hand landmarks detected by the pipeline. For each image in the dataset, 63 keypoints are extracted which will serve as raw features of the image.

Upon extracting the keypoints using MediaPipe, it is observed that not all images are converted into landmarks. Figure 5 summarizes the number of images per class that are converted and not converted by the pipeline. From this analysis, we divide the dataset into two kinds: Balanced and Imbalanced.

For the Balanced dataset, only 313 images are obtained from each of the classes which will be used for classification. Of the classes having more than 313 converted images, the 313 images used in the Balanced dataset are randomly selected. For the Imbalanced dataset, we perform classification on all images converted into landmarks. The number of images per class is observed in Fig. 5.

Performance evaluation

Figures 6A–6E show the box-and-whisker plots of the distribution of performance metrics obtained by the classifiers for the Balanced dataset. It can be observed that PHCA and SVM performed better than the other classifiers across all metrics since the interquartile ranges (IQRs) of the two classifiers lie significantly above than others. From Fig. 6E, it shows that the overall accuracy values of SVM ranges from 99.34% to 99.87%, with a mean of 99.45% across all trials. Meanwhile, PHCA’s overall accuracy ranges from 98.94% to 99.60%, with a mean of 99.31%. On the other hand, the maximum accuracy values of RF, KNN, LDA, and CART across the trials are 95.62%, 98.14%, 93.75%, and 96.81%, respectively. The same observations can be said for the other metrics as shown in Fig. 6A for the precision, Fig. 6B for recall, and Fig. 6C for F1-score. For the average specificity, the scale of difference in values among classifiers is smaller. SVM’s average specificity ranges from 99.97% to 99.99% with a mean of 99.97% while that of PHCA ranges from 99.95% to 99.98% with a mean of 99.97%. Compared to the average specificity values of RF, KNN, LDA, and CART which are 99.72%, 99.77%, 99.65%, and 99.81%, respectively. This implies that RF, KNN, LDA, and CART also performed well in terms of specificity. However, it can be noted that the distribution for specificity values are similar with the other metrics as observed from the box-and-whisker plots in Fig. 6D, which still indicates the superiority of PHCA and SVM. Figure 6F shows the average metric values across the trials of all classifiers. It provides additional evidence that SVM and PHCA performed better than the other classifiers, hence, supporting our claim.

Now, we can also deduce from Figs. 6A–6E that the performance, in terms of all metrics, of SVM and PHCA are not significantly different on the Balanced dataset. This is because for each metric, the median line (50% quartile) of SVM falls within the IQR of PHCA. For the precision values, the median of SVM is 99.48% and the IQR of PHCA is 99.22–99.49%. For the Recall values, the median of SVM is 99.46% and the IQR of PHCA is 99.19–99.47%. Similarly for the F1-score values, the median of SVM is 99.46% and the IQR of PHCA is 99.19–99.46%. For the specificity and accuracy values, the median of SVM is exactly the upper bound of the IQR of PHCA which are 99.97–99.98% and 99.20–99.47%, respectively. We investigate this claim further in the following section.

The IQRs of PHCA in all metrics are small relative to RF, KNN, LDA, and CART but is bigger than that of SVM. This implies that the dispersion of metric values is low, indicating consistent performance across the trials. Comparing this with KNN which has the largest IQR in each metric, implying higher dispersion in metric values. This suggests that the performance of KNN is highly dependent on the dataset and consequently, on its “n_neighbors” hyperparameter. The low dispersion of PHCA provides evidence to the stability of the classifier under changes in the dataset.

Similarly, Figs. 7A–7E show the box-and-whisker plots of the distribution of performance metrics obtained by the classifiers for the Imbalanced dataset. The superiority of PHCA and SVM to the other four classifiers in all setups can still be observed from the figure, based on the location of their IQRs. Figure 7E shows that the overall accuracy values of SVM ranges from 99.39% to 99.80%, with a mean of 99.60% across all trials. Meanwhile, PHCA’s overall accuracy ranges from 98.87% to 99.80%, with a mean of 99.40%. To compare, the maximum accuracy values of RF, KNN, LDA, and CART across the trials are 93.76%, 99.28%, 94.27%, and 97.24%, respectively. The scale of difference in the specificity values for the imbalanced dataset is also smaller. SVM’s average specificity ranges from 99.97% to 99.99% with a mean of 99.98% while that of PHCA ranges from 99.95% to 99.99% with a mean of 99.97%. Compared to the average specificity values of RF, KNN, LDA, and CART which are 99.73%, 99.97%, 99.75%, and 99.88%, respectively. Despite the excellent performance of these four classifiers in terms of specificity, the box-and-whisker plots in Fig. 7D still shows that PHCA and SVM performed better. The bar plots in Fig. 7F shows also that overall, PHCA and SVM performed better than the other classifiers.

Now, notice in Figs. 7A–7E that the median of SVM in all metrics did not fall within the IQRs of PHCA. This suggests that there is a significant difference between the two classifiers in all metrics. We further investigate this claim in the following section by using some statistical test.

Compared to the Balanced dataset, the IQRs of each classifier in all metrics also increased for the Imbalanced dataset. This means that the metric values of the classifiers are more dispersed, which is reasonable considering the nature of the dataset which makes classification more challenging. However, the IQRs of PHCA in all metrics are still smaller relative to RF, KNN, LDA, and CART and slightly bigger than SVM. This means that the dispersion of the metric values of PHCA is relatively low which indicates consistency in performance across trials.

Comparison of classification performance

Now, we empirically compare the classifiers in terms of overall performance, i.e., in terms of all metrics. For this, we state the null hypothesis that there is no significant difference in overall performance and hypothesize that there is. We use Friedman Test to test our claim.

The p-value obtained from the Friedman Test is less than the threshold value $\alpha =0.05$ in all trials for both the Balanced and Imbalanced datasets. Hence, we implement a post-hoc analysis using Nemenyi Test to pairwise compare the classifiers. The results are shown in Tables 2 and 3 for the Balanced and Imbalanced datasets, respectively. We focus on comparing PHCA with the other five classical classifiers.

From the tables, the results show that there is no significant difference in terms of overall performance between PHCA and SVM, but the performance of PHCA differs significantly from the other classifiers, except for KNN on some trials. This is true for both the Balanced and Imbalanced datasets. Our claim for the Balanced dataset in the previous section is supported by this result but not for the Imbalanced dataset. However, we remark that the claim in the previous section came from an educated guess based on the box-and-whisker plots. Our results from Nemenyi Test provide empirical evidence against this guess.

In some trials, KNN performed well and has shown similar performance with PHCA and SVM. But we note that the dispersion of metric values of KNN is high and this similarity of performance is not consistent across all trials. Hence, from the results in Figs. 6 and 7 and the result of comparison in Tables 2 and 3, we can conclude that PHCA and SVM performed best in all setups for both the Balanced and Imbalanced datasets and their is no significant difference in terms of their performance. This highlights that PHCA performance is at par with the best performing classifier for FSL recognition.

Misclassification analysis of PHCA

We further investigate the distribution of misclassified images by PHCA for the Balanced and Imbalanced datasets. Figure 8 shows the cumulative confusion matrix associated with the predictions obtained by PHCA.

PHCA made the most number of misclassifications on class M from the Balanced dataset, having a total of 11. Of these, most are classified into class F (6 images) and S (2). The second most misclassified by PHCA is class O which are incorrectly predicted to be in class C (2) and R (4). For the Imbalanced dataset, PHCA misclassified class X the most. Images from this class are often predicted to be in class T (4) and R (2). PHCA also misclassified a few images under class C which are usually incorrectly predicted to be in class O (3) and W (3). The classifier also mispredicted five images from class F to class B.

Shown in Fig. 9 are the images from the stated classes where PHCA had the most misclassifications. We can observe that the hand shape in class M are very similar to that of class S. Similarity in hand structure can be observed from class O and C; so are for class X and T. We also observe similar hand patterns from images under class B and F. These similarities in hand shape and structure show evidence for the misclassifications on the predictions of PHCA. Similar structures imply similar 3D coordinates corresponding to the landmarks obtained by MediaPipe, hence, resulting in the confusion of the classifier.

However, there are hand structures that are different but are still misclassified, such as for class M and F, class O and R, class X and R, and class C and W. These misclassifications may be due to the nature of the features obtained from the images.

Recall that MediaPipe extracts the 3D coordinates of the landmarks from the hand. However, not all images from the same class are consistent in terms of orientation, structure, and proximity with respect to the camera. This variability may have caused dissimilar hand structures to have similar coordinates. This result is also observed from Oliva et al. (2018). Their analysis shows that by using Cartesian coordinates, misclassification can occur only when the gesture is similar or there is a difference in position.

Additionally, PHCA performed at par, and sometimes better, than SVM in classifying images from particular classes. For classes C, F, H, I, M, R, and T, PHCA and SVM had the same number of misclassifications for the Balanced dataset. The two models also have the same number of misclassified images under class A, H, T, and W for the Imbalanced dataset. Meanwhile, PHCA had less number of misclassifications on class A, O, and X for the Balanced dataset. This provides evidence that on some cases, PHCA can perform better than SVM.