Isolated sign language recognition through integrating pose data and motion history images

Datasets

In this study, multiple sign language datasets are used to comprehensively evaluate the effectiveness of the proposed method. Each dataset serves a unique purpose in our experimental design, from method development to a comparative analysis with works in the literature.

Our experimental analysis is based on BosphorusSign22k-general, a subset of the BosphorusSign22k dataset (Camgoz et al., 2016; Özdemir et al., 2020). This subset contains 4,839 training videos and 949 test videos for a total of 174 sign word classes performed by six signers. The data of one signer are reserved as test data for signer-independent evaluation. This general subset provides a diverse representation of Turkish Sign Language and provides an ideal platform for the initial testing and refinement of our proposed method.

The complete BosphorusSign22k dataset contains 22,542 videos and 744 sign word classes. The test set is reserved for signer-independent evaluation. This dataset is used to evaluate the scalability of our method and its effectiveness on a larger set of sign words. Thus, the ability of the proposed method to understand sign languages in different contexts is tested in a wider framework.

In order to verify the applicability of our proposed method across different sign languages, we also consider the LSA-64 dataset, which contains 64 sign words from Argentine Sign Language. This dataset, created by Ronchetti, Quiroga & Lanzarini (2016), contains a total of 3,200 video images, each consisting of 10 signers repeating each word five times. The creators of the dataset did not specifically separate this dataset into training and testing sets. Therefore, based on studies in the literature, three different experimental sets were created in our study: Experiment 1 (E1), where the 5th and 10th signers are used as testing data; Experiment 2 (E2), where nine out of 10 signers are used for training data and the rest for testing data; and Experiment 3 (E3), where the dataset is randomly split into training data (80%) and testing data (20%).

Another dataset evaluated in our study is the GSL dataset, consisting of 310 Greek Sign Language words and created by Adaloglou et al. (2020). This dataset consists of a total of 40,826 videos generated by seven different signers. The data of one signer were excluded from the training data for signer-independent evaluations. Of the data excluded from the training data, 2,331 videos are reserved as validation data and 3,500 are reserved as test data.

Using these various datasets, we aim to provide a comprehensive validation of our proposed method, highlighting its robustness, versatility, and scalability and its potential for wide applicability in SLR and understanding tasks. A summary of the datasets used in our study is given in Table 2.

Pose extraction with MediaPipe

In the rapidly advancing fields of computer vision and machine learning, accurate human pose extraction from videos is crucial for applications such as SLR. A leading framework, MediaPipe’s Holistic solution (Lugaresi et al., 2019; Grishchenko & Bazarevsky, 2020), provides a comprehensive solution to efficiently track face, hand, and body positions for real-time, accurate pose estimation using deep learning. Our work is seamlessly integrated into our research for improved pose inference by leveraging MediaPipe’s Holistic solution. In this study, the reasons for choosing MediaPipe include its real-time processing capability, high accuracy rates, and extensive pose estimation features. MediaPipe effectively addresses the nuanced requirements encountered in SLR systems, thereby offering a high performance and low latency. Additionally, the fact that it is open-source, supported by a broad developer community, and continuously updated enhances the flexibility and adaptability of this technology.

In our study, first, using the MediaPipe Holistic solution, body, hand, and face pose landmark points were obtained from each video frame. A video ( $\mathrm{V}$) consists of $\mathrm{N}$ frames, denoted as $\left({\mathrm{I}}_1,{\mathrm{I}}_2,\dots ,{\mathrm{I}}_{\mathrm{N}}\right)$. Each frame has dimensions $\mathrm{W}\times \mathrm{H}\times \mathrm{C}$. W and H are the width and height of the frame, respectively, and C is the number of color channels. The MediaPipe Holistic model, denoted as M, processes each frame to output a set of landmarks ${\mathrm{L}}_{\mathrm{n}}$ for the face $({\mathrm{L}}_{\mathrm{f}\mathrm{a}\mathrm{c}\mathrm{e},\mathrm{n}})$, hands $({\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s},\mathrm{n}})$, and body $({\mathrm{L}}_{\mathrm{b}\mathrm{o}\mathrm{d}\mathrm{y},\mathrm{n}})$ (Eq. (1)):

(1) $${\mathrm{L}}_{\mathrm{n}}=\mathrm{M}\left({\mathrm{I}}_{\mathrm{n}}\right)=\left\{{\mathrm{L}}_{\mathrm{f}\mathrm{a}\mathrm{c}\mathrm{e},\mathrm{n}},{\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s},\mathrm{n}},{\mathrm{L}}_{\mathrm{b}\mathrm{o}\mathrm{d}\mathrm{y},\mathrm{n}}\right\}.$$

Here, ${\mathrm{L}}_{\mathrm{f}\mathrm{a}\mathrm{c}\mathrm{e},\mathrm{n}}$ contains 468 landmark points, ${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s},\mathrm{n}}$ contains 42 landmark points (21 for each hand), and ${\mathrm{L}}_{\mathrm{b}\mathrm{o}\mathrm{d}\mathrm{y},\mathrm{n}}$ contains 33 landmark points. Each landmark point $\mathrm{l}\in {\mathrm{L}}_{\mathrm{n}}$ is represented as a triple $\left(\mathrm{x},\mathrm{y},\mathrm{z}\right)$, where x and y are the pixel coordinates of the landmark in the image plane and z is the depth information relative to the camera plane. In the next subsection, the estimation of missing landmark points using linear interpolation and the generation of visual pose data from these points will be analyzed.

Completion of missing pose data with linear interpolation

If the hand is not in an appropriate position relative to the camera angle, if it moves out of the camera’s field of view, if the movements in the video are too fast, or if the hand is partially covered by an object or body during the video, MediaPipe may not be able to obtain the hand landmark points in some frames. Indeed, in the experiments with BosphorusSign22k-general, the dataset consists of 492,541 frames. In 21,551 of these frames, the right-hand pose landmark points were missing, and in 46,088 frames, the left-hand pose landmark points were missing. These missing landmark points may affect the classification performance.

In this study, we used linear interpolation to estimate missing hand pose data. Linear interpolation is a method used to estimate a new point between two known points. In the context of hand landmark data, imagine that there are $\mathrm{n}$ unknown hand landmark points between two known hand landmarks, and let $\mathrm{i}$ be the index of one of these unknown points. The value of a missing hand pose point, ${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g},\mathrm{n}}$, can be estimated using Eq. (2):

(2) $${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{m}\mathrm{i}\mathrm{s}\mathrm{s}\mathrm{i}\mathrm{n}\mathrm{g},\mathrm{n}}={\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{k}\mathrm{n}\mathrm{o}\mathrm{w}\mathrm{n},\mathrm{n}-1}+\mathrm{i}.{\displaystyle \frac{\left({\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{k}\mathrm{n}\mathrm{o}\mathrm{w}\mathrm{n},\mathrm{n}+1}-{\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{k}\mathrm{n}\mathrm{o}\mathrm{w}\mathrm{n},\mathrm{n}-1}\right)}{\left(\mathrm{n}+1\right)}.}$$

This formula allows us to interpolate the position of missing hand pose landmarks linearly by using the known positions of the hand landmarks immediately before and after the missing data, denoted as ${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{k}\mathrm{n}\mathrm{o}\mathrm{w}\mathrm{n},\mathrm{n}-1}$ and ${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d},\mathrm{k}\mathrm{n}\mathrm{o}\mathrm{w}\mathrm{n},\mathrm{n}+1}$. As an example, the HAND-POSE-MHI-1 data obtained from the incomplete pose data and the HAND-POSE-MHI-1 data obtained from the completed pose data are shown in Fig. 2.

Generation of visual pose data

After obtaining landmark points for the body, hands, and face from each video frame using the MediaPipe Holistic solution, visual pose data were created using the steps described in this subsection. Converting numerical landmark points into visual representations improves the interpretability of pose estimation in areas such as SLR, allowing for a more intuitive analysis of the captured poses. This visual pose data are shown in Fig. 3.

First, the BODY-POSE image was created. These images provide a comprehensive representation that visualizes the body, hands, and face in a single frame. This process is performed using all landmark points $\left({\mathrm{L}}_{\mathrm{n}}\right)$ and the links between them for each frame ( ${\mathrm{I}}_{\mathrm{n}}$) of the video ( $\mathrm{V}$). First, for each frame ( ${\mathrm{I}}_{\mathrm{n}}$), a blank black canvas ( ${\mathrm{C}}_{\mathrm{n}}$) of the same dimensions as the original image ( $\mathrm{W}\times \mathrm{H}$) is created. This canvas serves as a background for drawing both the landmark points and the connections between them. For each landmark point l ∈ L, a landmark point can be defined as $\mathrm{l}=\left(\mathrm{x},\mathrm{y},\mathrm{z}\right)$. Here, x and y denote the pixel coordinates of the landmark point on the canvas. We will not use the depth information z here, so we express a landmark point as $\mathrm{l}=\left(\mathrm{x},\mathrm{y}\right)$. The links between each landmark point represent the relationships between specific landmark pairs. Each link can be assigned a different color and thickness value. The drawing of the links is performed by drawing lines between the start and end landmark points. The drawing is done for each $\mathrm{l}\mathrm{i}\mathrm{n}{\mathrm{k}}_{\mathrm{k}}$, $\mathrm{l}\mathrm{i}\mathrm{n}{\mathrm{k}}_{\mathrm{k}}=\left({\mathrm{l}}_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}},{\mathrm{l}}_{\mathrm{e}\mathrm{n}\mathrm{d}}\right)$, using the start and end points defined as ${\mathrm{l}}_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}}=\left({\mathrm{x}}_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}},{\mathrm{y}}_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}}\right)$ and ${\mathrm{l}}_{\mathrm{e}\mathrm{n}\mathrm{d}}=\left({\mathrm{x}}_{\mathrm{e}\mathrm{n}\mathrm{d}},{\mathrm{y}}_{\mathrm{e}\mathrm{n}\mathrm{d}}\right)$. The drawing function is given below (Eq. (3)):

(3) $$\mathrm{D}\mathrm{r}\mathrm{a}\mathrm{w}\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\left({\mathrm{C}}_{\mathrm{n}},{\mathrm{l}}_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}},{\mathrm{l}}_{\mathrm{e}\mathrm{n}\mathrm{d}},\mathrm{c}\mathrm{o}\mathrm{l}\mathrm{o}{\mathrm{r}}_{\mathrm{k}},\mathrm{t}\mathrm{h}\mathrm{i}\mathrm{c}\mathrm{k}\mathrm{n}\mathrm{e}\mathrm{s}{\mathrm{s}}_{\mathrm{k}}\right).$$

It draws a line on the canvas ( ${\mathrm{C}}_{\mathrm{n}}$) between the points ${\mathrm{l}}_{\mathrm{s}\mathrm{t}\mathrm{a}\mathrm{r}\mathrm{t}}$ and ${\mathrm{l}}_{\mathrm{e}\mathrm{n}\mathrm{d}}$, with the specified $\mathrm{c}\mathrm{o}\mathrm{l}\mathrm{o}{\mathrm{r}}_{\mathrm{k}}$ and $\mathrm{t}\mathrm{h}\mathrm{i}\mathrm{c}\mathrm{k}\mathrm{n}\mathrm{e}\mathrm{s}{\mathrm{s}}_{\mathrm{k}}$. This function provides a visual representation of both the points and the connections between them and is the basis of the visualization process.

In addition to BODY-POSE images, our study focused on the detailed analysis of hand gestures, a critical component of sign language. For this purpose, HAND-POSE images were produced, showing only hand poses. These images were drawn using only the landmark points $({\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s}})$, with steps similar to the BODY-POSE generation steps.

After obtaining the BODY-POSE and HAND-POSE data, focus area cropping was performed on these images. This process aims to create a standardized structure to be used in subsequent analysis phases by cropping the unnecessary parts of the visual data. First, we identify the bounding boxes for the data to be cropped. For these bounding boxes, we use a set of ${\mathrm{L}}_{\mathrm{n}}$ landmark points obtained over the whole video. For the given set of ${\mathrm{L}}_{\mathrm{n}}$ landmark points, that is, for each landmark point ${\mathrm{l}}_{\mathrm{i}}=\left({\mathrm{x}}_{\mathrm{i}},{\mathrm{y}}_{\mathrm{i}}\right)$, the process of finding the minimum and maximum values of x and y between $\mathrm{i}=1,2,\dots ,\mathrm{N}$ is mathematically shown in Eqs. (4)–(7):

(4) $${\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}={\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{x}}_{\mathrm{i}},$$

(5) $${\mathrm{y}}_{\mathrm{m}\mathrm{i}\mathrm{n}}={\mathrm{m}\mathrm{i}\mathrm{n}}_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{y}}_{\mathrm{i}},$$

(6) $${\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}={\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{x}}_{\mathrm{i}},$$

(7) $${\mathrm{y}}_{\mathrm{m}\mathrm{a}\mathrm{x}}={\mathrm{m}\mathrm{a}\mathrm{x}}_{\mathrm{i}=1}^{\mathrm{N}}{\mathrm{y}}_{\mathrm{i}}.$$

After finding the minimum and maximum values of the x and y coordinates, we add a padding value to leave enough space for the landmark points and their connections to be fully visible in the image. During this process, we check that the minimum values are not less than 0 and that the maximum values are not greater than the height ( $\mathrm{H}$) and width ( $\mathrm{W}$) of the image. These corrections guarantee that the bounding box does not exceed the area of the image and contains all relevant landmarks. The operations are shown in Eqs. (8)–(11):

(8) ${\text{x}}_{\text{min}}^{\prime }=\max \left(0,{\text{x}}_{\text{min}}-\text{padding}\right),$

(9) $${\mathrm{y}}_{\mathrm{m}\mathrm{i}\mathrm{n}}^{\mathrm{\prime }}=max\left(0,{\mathrm{y}}_{\mathrm{m}\mathrm{i}\mathrm{n}}-\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\right),$$

(10) $${\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}^{\mathrm{\prime }}=min\left(\mathrm{W},{\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}+\mathrm{p}\mathrm{a}\mathrm{d}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g}\right),$$

(11) ${\text{y}}_{\text{max}}^{\prime }=\min \left(\text{H},{\text{y}}_{\text{max}}+\text{padding}\right).$

After this process, we make the bounding box approach the square format. To do this, we first find the width ( ${\mathrm{W}}^{\mathrm{\prime }}$) and height ( ${\mathrm{H}}^{\mathrm{\prime }}$) of the bounding box, as shown in Eqs. (12) and (13):

(12) $${\mathrm{W}}^{\mathrm{\prime }}={\mathrm{x}}_{\mathrm{m}\mathrm{a}\mathrm{x}}^{\mathrm{\prime }}-{\mathrm{x}}_{\mathrm{m}\mathrm{i}\mathrm{n}}^{\mathrm{\prime }},$$

(13) $${\mathrm{H}}^{\mathrm{\prime }}={\mathrm{y}}_{\mathrm{m}\mathrm{a}\mathrm{x}}^{\prime }-{\mathrm{y}}_{\mathrm{m}\mathrm{i}\mathrm{n}}^{\prime }.$$

If the width of the bounding box is greater than its height, we need to expand the box around the height axis; otherwise, we need to expand the box around the width axis. The amount of expansion (Δ) we will use for this operation is calculated using Eq. (14):

(14) $$\mathrm{\Delta }={\displaystyle \frac{\left|{\mathrm{W}}^{\mathrm{\prime }}-{\mathrm{H}}^{\mathrm{\prime }}\right|}{2}.}$$

Once the expansion amount ∆ is found, Eq. (15) is used to expand the bounding region:

(15) $\{\begin{array}{cc}{\text{y}}_{\text{min}}^{\prime }=\max \left(0,{\text{y}}_{\text{min}}-\Delta \right),& {\text{ y}}_{\text{max}}^{\prime }=\min \left(\text{H},{\text{y}}_{\text{max}}+\Delta \right)\text{\hspace{0.17em}\hspace{0.17em}\hspace{0.17em}if\hspace{0.17em}}{\text{W}}^{\prime }>{\text{H}}^{\prime },\\ {\text{x}}_{\text{min}}^{\prime }=\max \left(0,{\text{x}}_{\text{min}}-\Delta \right),& {\text{ x}}_{\text{max}}^{\prime }=\min \left(\text{W},{\text{x}}_{\text{max}}+\Delta \right)\text{\hspace{0.17em}\hspace{0.17em}otherwise}.\end{array}$

This equation provides the coordinate information necessary to proportionally adjust both the width and length of the bounding region without exceeding the physical boundaries of the image. The cropping process shown in Eq. (16) is then performed for each frame of the video (V):

(16) ${\text{I}}_n^{\prime }={\text{I}}_{\text{n}}\left[{\text{y}}_{\text{min}}^{\prime }:{\text{y}}_{\text{max}}^{\prime },{\text{x}}_{\text{min}}^{\prime }:{\text{x}}_{\text{max}}^{\prime }\right].$when applied to each frame of the video (V), this process ensures that all frames in the video are cropped in a coherent and focused manner. As a result, this process focuses on the area of the pose images in the original image and eliminates redundant data. When applied to each frame of the video (V), this process ensures that all frames within the video are cropped in a coherent and focused manner. The result is a focus on the area of the pose images in the original image and the elimination of redundant data.

After obtaining the BODY-POSE and HAND-POSE data, two new features were created using these data: FACE-POSE and MERGED-HAND-POSE. FACE-POSE is obtained by cropping the face region from the BODY-POSE data with the help of ${\mathrm{L}}_{\mathrm{f}\mathrm{a}\mathrm{c}\mathrm{e}}$ landmark points. MERGED-HAND-POSE is obtained from the HAND-POSE data by cropping the regions corresponding to the hands through ${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s}}$ landmark points and then merging them. These cropped features provide a closer look and deeper insight into the subtleties of human poses. While the creation of FACE-POSE data is important for the focused analysis of facial expressions and nuances, similarly, MERGED-HAND-POSE data provide a richer context for the detailed interpretation of hand gestures and positions. These features are expected to influence the recognition accuracy in the SLR system.

Motion history image

A motion history image (MHI) is a representation method used to summarize motion information from video frames in a single image (Bobick & Davis, 2001). MHIs are widely used in motion analysis and motion-based classification applications. They are especially important in areas such as human interactions, gestures, and SLR (Zhang et al., 2019; Naeem et al., 2020; Chandragiri & Ijjina, 2021; Sincan & Keles, 2021; Ghosh et al., 2023).

The fundamental concept underlying MHIs is to visually represent the temporal progression of motion within a sequence of video frames. A weighted combination of every frame from the start to the end of the motion is required for this. The outcome of the MHI algorithm is an image, where the intensity of each pixel is calculated according to the temporal proximity of the motion at that pixel (Ahad, 2013).

The calculation of the MHI starts by taking consecutive differences of video frames. These differences indicate where and when the motion occurred. Then, this motion information is displayed in an MHI that accumulates across all frames. The MHI not only provides information about the location and time of the movement but also captures features such as the direction and speed of the movement. This is useful in situations where the dynamic properties of motion are important for classification (Ahad et al., 2012).

The mathematical formula of the MHI is expressed in Eq. (17):

(17) $${\mathrm{H}}_{\tau }\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)=\{\begin{array}{cc}\tau & \mathrm{i}\mathrm{f}\psi \left(\mathrm{x},\mathrm{y},\mathrm{t}\right)=1,\\ max\left(0,{\mathrm{H}}_{\tau }\left(\mathrm{x},\mathrm{y},\mathrm{t}-1\right)-\delta \right)& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\end{array}$$

In this formula, $\left(\mathrm{x},\mathrm{y}\right)$ represents pixel positions, and $\mathrm{t}$ represents time. The parameter $\tau$ stands for the travel time of a frame, and $\delta$ is called the decay parameter. $\psi$ is the update function describing the presence of motion. This update function (Eq. (18)) is applied every time a new video frame is analyzed:

(18) $$\psi \left(\mathrm{x},\mathrm{y},\mathrm{t}\right)=\{\begin{array}{cc}1& \mathrm{i}\mathrm{f}\mathrm{D}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)>\xi ,\\ 0& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e},\end{array}$$where $\xi$ is a threshold parameter. $\mathrm{D}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)$ represents the difference between two consecutive frames, which is defined in Eq. (19):

(19) $$\mathrm{D}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)=\left|\mathrm{I}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)-\mathrm{I}\left(\mathrm{x},\mathrm{y},\mathrm{t}-1\right)\right|.$$

As a result of this process, pixels that have recently moved create a brighter image, while older movements create a darker image. As a result, the MHI is a grayscale image that visualizes motion in a video sequence.

Beyond the grayscale MHI, the RGB-MHI (Sincan & Keles, 2021) builds on the foundations of MHIs, extending the traditional MHI to provide a richer and more detailed representation of motion dynamics over time. An RGB-MHI is constructed by dividing the video sequence into three equal parts and computing separate motion histories for each. Each color channel—red, green, and blue—is assigned to represent motion information at specific time intervals. The calculation of the RGB-MHI can be considered an extension of the MHI calculation and is adapted to process and compile motion data across RGB channels.

In this work, we apply the MHI and RGB-MHI methods to the BODY-POSE, HAND-POSE, and FACE-POSE data obtained in the previous subsection. The normal MHI and RGB-MHI are called MHI-1 and MHI-3, respectively, to indicate the number of channels. For a comparative analysis with these generated MHI data, we also extract MHI-1 and MHI-3 features from the RAW video data. The MHI-1 and MHI-3 features of an example sign are shown in Fig. 4.

Finger pose-based MHI

This subsection elucidates the development of the finger pose-based motion history image (FP-MHI) feature, a novel contribution of our research aimed at capturing the temporal dynamics of finger movements within SLR systems. Initially, visual pose data for each finger are meticulously generated by employing landmarks specific to each finger within ${\mathrm{L}}_{\mathrm{h}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{s},\mathrm{n}}$, following the procedures outlined from Eqs. (3) to (16). This methodological approach results in the creation of five distinct image sequences, each providing a depiction of finger movements across a video frame. Subsequently, the MHI technique is individually applied to these sequences, yielding a dedicated MHI dataset for each finger. Integrating these datasets culminates in the formation of a comprehensive five-channel FP-MHI image, encapsulating the essence of finger motion dynamics. The mathematical formulation of the FP-MHI is delineated in Eq. (20):

(20) $$\mathrm{F}\mathrm{P}-\mathrm{M}\mathrm{H}\mathrm{I}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({\mathrm{H}}_{\tau }^{\mathrm{T}\mathrm{H}\mathrm{U}\mathrm{M}\mathrm{B}}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right),{\mathrm{H}}_{\tau }^{\mathrm{I}\mathrm{N}\mathrm{D}\mathrm{E}\mathrm{X}}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right),{\mathrm{H}}_{\tau }^{\mathrm{M}\mathrm{I}\mathrm{D}\mathrm{D}\mathrm{L}\mathrm{E}}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right),{\mathrm{H}}_{\tau }^{\mathrm{R}\mathrm{I}\mathrm{N}\mathrm{G}}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right),{\mathrm{H}}_{\tau }^{\mathrm{P}\mathrm{I}\mathrm{N}\mathrm{K}\mathrm{Y}}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)\right).$$

In this equation, for each finger, the MHI, denoted as ${\mathrm{H}}_{\tau }^{\mathrm{f}}\left(\mathrm{x},\mathrm{y},\mathrm{t}\right)$, represents the accumulation over time ( $\mathrm{t}$) of the motion of the finger f at a given $\mathrm{x},\mathrm{y}$ pixel location, and the Concat function represents the process of merging the given MHIs, resulting in a 5×x×y FP-MHI that presents the accumulation over time of the motions of five different fingers in five different channels. In sign language classification, this FP-MHI is a crucial step in capturing more subtle gestures and meanings. This is an approach to the detailed analysis of finger gestures that has not been previously discussed in the literature. By employing this approach, the temporal progression of gesture data can be represented with greater accuracy, thereby potentially enhancing the precision of sign language classification. An example of the generation of FP-MHI data is shown in Fig. 5.

Other preprocessing steps

A sample representation of the data that will be used for our final SLR system based on the processes described above is shown in Fig. 6. These data undergo a further series of preprocessing steps before they are given as input to the ResNet-18 models.

First, in order for the deep learning model used in our research to train efficiently and make accurate predictions, the input data must have a certain format and size. For this reason, all data were reduced to a 112 × 112 resolution, which is the size expected by the model as input. However, since the MERGED-HAND-POSE data contain the combined image of both hands, the size was set to 56 × 112.

Second, visual pose data are image sequences consisting of multiple frames that capture the evolution of motion over time. Therefore, for our ResNet-18 model to process these data effectively, we need to select a certain number of frames from each video sequence that the model can process. To increase the diversity and generalizability of the model training, we randomly select 32 frames from each video sequence in each training cycle. In this way, the model is able to learn the motions from different time periods and recognize the general motion pattern. However, in order to ensure temporal consistency and comparability in our dataset, we have selected equally spaced frames from each video sequence to evaluate and test the performance of our model. This method allows us to more fairly and consistently evaluate the performance of the model across different videos.

In the last stage, the normalization process was performed. This process is formulated in Eq. (21):

(21) $$\mathrm{N}\mathrm{o}\mathrm{r}\mathrm{m}\mathrm{a}\mathrm{l}\mathrm{i}\mathrm{z}\mathrm{e}\mathrm{d}\left(\mathrm{x}\right)={\displaystyle \frac{\mathrm{x}-\mu }{\sigma }.}$$

Here, x is the original pixel value, $\mu$ represents the mean, and $\sigma$ represents the standard deviation. The normalization is performed to scale the input data and allow these data to be processed more efficiently by the model.

The data sizes after all preprocessing steps to be given as input to the ResNet-18 models for our final SLR system are shown in Table 3.

ResNet-18 model

In our study, the ResNet-18 model (He et al., 2016) has been selected as the foundational model for the SLR system. This choice was made considering ResNet-18’s ability to provide effective training in a short time and due to the fact that its structural features require relatively low computational costs. The skip connection architecture of the model reduces the vanishing gradient problem encountered in training deeper networks while enhancing the model’s learning capacity. This feature is particularly valuable when working with limited datasets, as it strengthens the model’s ability to generalize from less data (Oyedotun, Ismaeil & Aouada, 2023).

During the adaptation of ResNet-18 for the SLR task, the architecture of the model was fine-tuned using pre-defined weights that had been trained on the ImageNet dataset. The transfer learning approach allows the model to quickly adapt to our sign language dataset, significantly speeding up the training process (Shaha & Pawar, 2018). This methodology aims to improve the performance in the new task, thereby enhancing the accuracy of sign language classification.

The structure of the model starts with a convolution layer with 64 filters with three channels (red, green, blue) for RGB images in the input layer. However, the data used in this study can have different sizes when given as input to our model: one channel (1D) for MHI-1, three channels (3D) for MHI-3 (RGB-MHI), five channels (5D) for finger pose MHI, and thirty-two channels (32D) for video pose data. Since the dimensions of these data are not compatible with the default input dimensions of ResNet-18, an appropriate preprocessing step was performed. Specifically, a convolution layer was added at the beginning of the model architecture with an input dimension varying depending on the type of data, an output dimension of 3, a 3 × 3 kernel size, and a step size of 1. This preprocessing step is of critical importance, as it transforms data of various sizes into a standard format that can be effectively processed by the ResNet-18 model. Such a transformation is essential for maintaining the standard dimensions of the data without hindering the model’s ability to leverage its pre-trained weights for feature extraction.

Following the adapted input layer, the model incorporates a maximum pooling layer that effectively reduces the size of each feature map. Subsequently, four layers that constitute the model’s depth are stacked, with each layer containing two residual blocks. These residual blocks assist in mitigating potential performance degradation when deeper architectures of the model are trained.

By default, the rectified linear unit (ReLU) activation function is used in the ResNet-18 model architecture. The formula of this activation function is shown in Eq. (22):

(22) $$\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{x}\right)=\mathrm{m}\mathrm{a}\mathrm{x}\left(0,\mathrm{x}\right).$$

However, in the context of this study, the default activation function in the ResNet-18 architecture, ReLU, has been replaced with the randomized leaky rectified linear unit (RReLU). RReLU introduces non-parametric behavior by randomly selecting a slope for negative activation values within predefined lower and upper bounds. This approach enhances the model’s capacity to learn and generalize, significantly improving its robustness against overfitting (Xu et al., 2015). The RReLU activation function can be mathematically represented as follows:

(23) $$\mathrm{R}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left(\mathrm{x}\right)=\{\begin{array}{cc}\mathrm{x}& \mathrm{i}\mathrm{f}\mathrm{x}\ge 0,\\ \mathrm{a}\mathrm{x}& \mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\end{array}$$

In this formula, $\mathrm{a}$ is a coefficient that is randomly drawn from a uniform distribution between the specified lower (default:1/8) and upper (default:1/3) bounds during the training phase. This randomness introduces variability into the activation, which contributes to the model’s ability to generalize better and prevent overfitting (Banerjee, Mukherjee & Pasiliao, 2020). In the “Experimental Studies” section, the effect of using the RReLU activation function instead of ReLU is analyzed.

Fusing of extracted features

In order to perform the SLR task in our study, a separate pre-trained ResNet-18 model was trained for each feature obtained from the preprocessing stage, and after the training processes were completed, the features obtained by from the flattening layers of models were extracted to be used in the fusion and classification processes.

The fusion of features was accomplished through a concatenation operation, a method chosen for its simplicity and effectiveness in preserving the integrity and distinctiveness of each feature set. Suppose that we have $\mathrm{n}$ feature vectors ${\mathrm{v}}_1,{\mathrm{v}}_2,\dots ,{\mathrm{v}}_{\mathrm{n}}$, each with a dimensionality of 512, extracted from the flattening layers of the pre-trained ResNet-18 models. The concatenated feature vector ${\mathrm{v}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}$ is then formulated as follows (Eq. (24)):

(24) $${\mathrm{V}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}=\mathrm{C}\mathrm{o}\mathrm{n}\mathrm{c}\mathrm{a}\mathrm{t}\left({\mathrm{v}}_1,{\mathrm{v}}_2,\dots ,{\mathrm{v}}_{\mathrm{n}}\right).$$

This concatenated vector ${\mathrm{V}}_{\mathrm{f}\mathrm{u}\mathrm{s}\mathrm{e}\mathrm{d}}$ amalgamates the individual strengths of each feature set, thereby providing a holistic representation of the different gesture data that is both comprehensive and detailed. The primary advantage of this feature fusion strategy lies in its capacity to enhance the classifier’s understanding and interpretation of the data. By amalgamating features derived from various aspects of the sign language gestures, the fused feature vector offers a richer and more nuanced representation of the data. This, in turn, facilitates a more accurate and robust classification performance, as the classifier can leverage a broader spectrum of information to discern between different sign language gestures.

Support vector machine

In this study, the support vector machine (SVM) is used to classify the fused features. The SVM is a supervised learning algorithm that aims to identify the optimal hyperplane for separating the feature space (Cortes & Vapnik, 1995). During the training process, the SVM constructs hyperplanes within a high-dimensional space in order to effectively partition the training dataset into distinct classes. In cases in which the feature set is not linearly separable, a kernel function is employed to facilitate the transformation of the data into a new vector space. The present study employed radial basis function (RBF) kernels. The RBF kernel is a kernel that utilizes a radial (exponential) function to smooth a distance measure (Amari & Wu, 1999). In contrast to linear kernels, this particular kernel possesses the ability to effectively address the nonlinear nature of the association between class labels and attributes.

Evaluation metrics

This research utilizes a variety of evaluation metrics to gauge the effectiveness of the proposed methodology. The key definitions for these metrics are as follows:

• True positives (TP): These are cases where the model accurately identifies an example as being part of a particular class.

• False positives (FP): These cases occur when the model wrongly labels an example as belonging to a certain class when it does not belong to that class.

• False negatives (FN): These happen when the model fails to recognize actual examples of a class, incorrectly marking them as not belonging to that class.

• True negatives (TN): These are instances where the model correctly identifies an example as not belonging to a specific class.

The overall effectiveness of the model across different categories is measured by the accuracy (Eq. (25)). This is calculated as the ratio of the number of correctly identified examples to the overall number of examples assessed. For a more nuanced evaluation, the precision and recall metrics are used. The precision (Eq. (26)) is the proportion of accurately identified positive examples out of all examples identified as positive for a specific class. The recall (Eq. (27)) measures how many actual positive examples the model correctly identifies. The F1-score (Eq. (28)) is a combined metric, representing the harmonic mean of the precision and recall and offering a balanced view of these two measures. The formulas are given below:

(25) $$\mathrm{A}\mathrm{c}\mathrm{c}\mathrm{u}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{y}={\displaystyle \frac{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{C}\mathrm{o}\mathrm{r}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{t}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}}{\mathrm{T}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}\mathrm{N}\mathrm{u}\mathrm{m}\mathrm{b}\mathrm{e}\mathrm{r}\mathrm{o}\mathrm{f}\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{d}\mathrm{i}\mathrm{c}\mathrm{t}\mathrm{i}\mathrm{o}\mathrm{n}\mathrm{s}},}$$

(26) $$\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{P}},}$$

(27) $$\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}={\displaystyle \frac{\mathrm{T}\mathrm{P}}{\mathrm{T}\mathrm{P}+\mathrm{F}\mathrm{N}},}$$

(28) $$\mathrm{F}1-\mathrm{S}\mathrm{c}\mathrm{o}\mathrm{r}\mathrm{e}=2\times {\displaystyle \frac{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}\times \mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}{\mathrm{P}\mathrm{r}\mathrm{e}\mathrm{c}\mathrm{i}\mathrm{s}\mathrm{i}\mathrm{o}\mathrm{n}+\mathrm{R}\mathrm{e}\mathrm{c}\mathrm{a}\mathrm{l}\mathrm{l}}.}$$

For a comprehensive evaluation, the macro-average of the precision, recall, and F1-score for each class is calculated. This approach gives equal weight to each category’s contribution to the model’s overall effectiveness, ensuring a balanced assessment across all classes.

Classification results of pose images

Initially, the ResNet-18 models were trained using raw video data and four different types of pose data generated from these data. In particular, this experiment compares the performance of the models trained with raw video data with those trained with pose data to compare the accuracy of SLR for each feature. The effect of using the RReLU activation function instead of ReLU is also demonstrated.

According to Fig. 7, the comparison between the model activation functions shows that ResNet-18 models trained with RReLU show a higher classification accuracy than the model trained with ReLU for all five features. These results confirm that the activation function can play a critical role in the performance of deep learning models.

When the classification results obtained are analyzed, HAND-POSE show higher accuracy rates than the other categories. This indicates that hand gestures play a critical role in sign language classification. This apparent success in HAND-POSE shows that the essence of sign language is largely based on gestures made with the hands.

The MERGED-HAND-POSE feature contains more details of the hand shape, orientation, and so on. This detail is achieved by cropping and merging the hand regions from the main image. This process allowed the hand regions, which are normally smaller and indistinct in the overall image, to be visually represented in a larger and more detailed structure for the MERGED-HAND-POSE feature. However, despite this detailed information, the MERGED-HAND-POSE feature does not include the position information of the hands that is available in the HAND-POSE feature, which is an important deficiency in the classification process. Knowledge of the position of the hands plays a critical role in the classification of isolated sign language words, as some signs are communicated through position changes. Therefore, the loss of this information by the MERGED-HAND-POSE feature leads to a significant decrease in the classification accuracy. These findings show how critical hand position information is in the selection of features to be used in SLR systems.

BODY-POSE features are critical for sign language classification. These features cover the general movements and positions of the body and how other parts of the body interact with hand movements. However, the accuracy of the BODY-POSE feature was slightly lower than that of the hand features (HAND-POSE and MERGED-HAND-POSE). The main reason for this is that hand movements and positions have a more decisive role in determining the meaning of sign language words. Detailed movements of the hands can directly affect the meaning of many sign language words, whereas general body movements have an indirect role in this interpretation. Additionally, since the BODY-POSE features cover a larger area of the body, detailed hand movements are less prominent in this larger area. This could be another factor affecting the classification accuracy of body features.

BODY-POSE features, which are direct pose representations of raw video data, showed a higher classification accuracy than raw video data. This demonstrates the advantage of pose data over raw images. This is because pose data focus only on the motion or gesture and largely eliminate the variations that may arise from the background and the signer’s physical features, such as their hair, beard/mustache, clothes, or clothing style. Therefore, these features increase the generalizability of the pose data and make these data superior to raw video data in classification.

The low accuracy of the FACE-POSE classification indicates that facial features alone are not fully sufficient for sign language classification. However, sign language is not limited to hand gestures; facial expressions and body language also play an important role. Therefore, the integrated use of facial features in the model is critical to achieving more accurate classifications.

Classification results of POSE-MHI features

In this subsection, we evaluate the classification performance of ResNet-18 with different MHI features. The findings presented in Fig. 8 reflect the test accuracies obtained with various MHI features. This investigation is part of our goal to explore how effectively MHI features can capture temporal information in sign language videos, alongside the spatial features evaluated in the previous subsection.

The findings presented in Fig. 8 demonstrate that ResNet-18 models trained using the RReLU activation function exhibit superior accuracy rates for MHI features compared to models trained with the ReLU activation function. This finding corroborates the findings observed in the analysis of pose data. It is conceivable that RReLU increases the learning capacity of the model, allowing it to obtain more generalizable and recognizable features.

The results suggest that pose-based MHI features, including BODY-POSE-MHI-1, BODY-POSE-MHI-3, HAND-POSE-MHI-1, HAND-POSE-MHI-3, and FP-MHI, exhibit higher accuracy rates compared to MHI features derived from unprocessed video data, namely RAW-MHI-1 and RAW-MHI-3. The potential benefits associated with the use of pose-based MHI features in the context of SLR are demonstrated by the findings of this study. Furthermore, this research provides the opportunity to perform a comparative analysis between single-channel features and three-channel MHI features. Based on the findings, it can be concluded that the use of three-channel MHI (MHI-3) features yields a more accurate result compared to the implementation of single-channel MHI (MHI-1) features. The MHI-1 technique generates a single representation that covers all motion over the entire video duration, whereas the MHI-3 strategy divides the temporal interval into three different phases. This increases the granularity of the data available for the classification and analysis of sign language.

The proposed feature, known as the FP-MHI, exhibited the highest level of accuracy compared to the other features examined, achieving an impressive rate of 91.41%. This superior performance of the FP-MHI is due to its capacity to provide discriminative information for sign language words. The role of finger positions and movements is of the utmost importance in the communication process of sign language. This suggests that the accurate recognition and classification of finger movements and positions can significantly improve the overall SLR performance.

In the following subsections of the article, the proposed system is implemented using the ResNet-18 model with the RReLU activation function. Furthermore, MHI-3 features are used, together with the pose data and FP-MHIs.

Effect of filling in missing poses with linear interpolation

Although the performance of MediaPipe is generally satisfactory, some pose data may not be accurately estimated due to fast hand movements or occlusion. In order to overcome such shortcomings, we chose to complete the missing pose data using linear interpolation when performing feature extraction based on hand poses. The classification results based on the completed features are presented in Fig. 9. In addition, in order to evaluate the effectiveness of linear interpolation, the performance results obtained with incomplete pose data are also presented in the same figure.

According to the results in Fig. 9, it can be seen that the completion of missing pose data using linear interpolation leads to an overall increase in the classification performance. Missing data can cause the model to stumble, especially when it is analyzing dynamic movements and rapid changes. Therefore, filling the gaps using linear interpolation allows the model to classify such motions more accurately. Furthermore, these results reveal how effective even simple and computationally efficient methods such as linear interpolation can be in solving the missing data problem. This is especially important when working with large datasets and complex model structures, as the application of such methods can save both time and resources.

Fusion and classification results of hand properties

In this subsection, we fuse the features extracted from the flatten layers of the ResNet-18 models trained using HAND-POSE, MERGED-HAND-POSE, HAND-POSE-MHI-3, and FP-MHI in various combinations and classify them with the SVM. The results of these operations are presented in Table 4. This approach is based on the hypothesis that combining hand pose data with hand gestures, which are captured through MHIs and encompass both static and dynamic movements, can potentially improve the SLR accuracy by providing a more robust set of features. The need for such a fusion stems from the complex nature of sign language, which often requires a multi-faceted feature set to accurately interpret the rich variety of hand gestures.

Table 4 illustrates the impact of different combinations of hand features on not only the classification accuracy but also the precision, recall, and F1-score. While certain hand features alone demonstrate a high classification accuracy, the fusion of these features plays a crucial role in elevating the overall performance. Notably, the accuracy of 92.83% achieved by combining HAND-POSE and HAND-POSE-MHI-3 is significantly enhanced to 94.62% with the addition of MERGED-HAND-POSE. However, the inclusion of FP-MHIs proves to be decisive in further improving the performance, culminating in a classification accuracy of 95.36%, along with corresponding increases in the precision, recall, and F1-score. This enhancement underscores the importance of integrating FP-MHIs with other hand features, markedly improving the classification accuracy and the overall robustness of the model.

The in-depth examination of hand characteristics is vital in the categorization of sign language. The findings from this research corroborate the idea that hand features, and their various combinations, have a substantial impact on the accuracy, precision, recall, and F1-score of the algorithms employed in sign language classification, emphasizing the importance of a comprehensive feature set for effective interpretation.

Impact of non-manual features on classification performance

In sign language, not only the position and movements of the hands but also body and facial movements are an important part of communication. In the realm of communication, facial expressions and body language are not only supplementary but play a pivotal role in conveying meanings and categorizing concepts. Within this framework, our study delves into the integration of hand features with body and facial expressions and examines the impact of these combinations on the accuracy of the classification. The results stemming from the amalgamation of hand features have been previously discussed. Here, we analyze the results of integrating BODY-FEATURES and FACE-FEATURES in different combinations. BODY-FEATURES are obtained by fusing BODY-POSE and BODY-POSE-MHI-3 features, and FACE-FEATURES are obtained by fusing FACE-POSE and FACE-POSE-MHI-3 features.

Table 5 shows the fusion of non-manual features with hand features and their classification results using the SVM in the BosphorusSign22k-general dataset. The table includes the test accuracy, precision, recall, and F1-score for different feature combinations.

The analysis of the table reveals that the classification accuracy for hand features alone is 95.36%, with corresponding precision, recall, and F1-score values of 94.77%, 95.44%, and 94.29%, respectively. When the BODY and FACE features are used independently, their accuracies are 88.61% and 14.33%, with the fusion of the HAND and BODY features increasing the accuracy to 95.89%. This combination achieves a slightly higher accuracy, precision, recall, and F1-score compared to the HAND and FACE feature combination, which results in an accuracy of 95.57%, a precision of 94.98%, a recall of 95.99%, and an F1-score of 94.63%. Using only the BODY and FACE features yields an accuracy of 86.93%, the highest achieved without incorporating HAND features. The highest classification accuracy of 96.94%, along with a precision of 96.43%, recall of 96.99%, and F1-score of 96.31%, is achieved by combining all the HAND, BODY, and FACE features. The fusion of HAND features with BODY and FACE features is crucial for attaining a higher accuracy in sign language classification. However, the contribution of the facial features alone to accuracy is relatively low, suggesting that facial gestures are more effective when they are combined with other features rather than being used independently.

Comprehensive performance analysis of the proposed method

In our study, we conducted a detailed performance analysis of our proposed SLR method, focusing on the precision, recall, and F1-scores for various signs and examining misclassification examples. This comprehensive evaluation is important to understand the effectiveness of the model and areas for improvement.

Table 6 shows the high performance of our model in correctly identifying most sign language words. This highlights the model’s strong capability to accurately recognize a diverse range of signs, showcasing its robustness and reliability. However, certain words, such as “early,” “address,” and “come,” exhibit a lower precision, underscoring specific challenges within the model. These may stem from the need for more varied training data or improvements in the feature extraction processes. In addition, the method generally retains high recall scores for signs with low precision. This suggests that although the model may confuse certain signs with others, it is still able to reliably detect their presence.

Table 7 provides insights into the model’s behavior and limitations through its detailed confusion analysis. It shows how the model confuses similar signs, such as “address” and “business address” or “children” and “child,” indicating challenges in distinguishing subtle differences. This could be due to similarities in the gesture dynamics, hand shapes, or movement patterns inherent to these sign pairs. These findings emphasize the importance of enriching the training dataset with a wider range of examples, in particular focusing on physically similar signs.

In summary, our analysis comprehensively showcases the strong aspects of our model for SLR. This study not only emphasizes the capabilities of our current model but also provides valuable contributions to the ongoing research in the field of SLR and interpretation. The development of training strategies and feature extraction techniques that will further enhance the model’s performance represents significant steps in advancing this field.

Temporal and cost analysis of the proposed method

In this subsection, we perform a temporal and cost analysis of our proposed SLR system. These analyses are performed on hardware with an Intel Core i5-8400 processor, 16 GB RAM, and a GeForce GTX 1080 Ti GPU, which was also used in the design of the system. We first compare the ResNet-18 model used in the proposed system with other efficient state-of-the-art models, and then we evaluate the inference time performance of each component used in our proposed final SLR system. We also perform some optimizations to improve the temporal performance and examine the inference time using the entire test set.

We first compare the performance of the ResNet-18 model used in our proposed SLR system with a set of state-of-the-art models with parameters equal to or less than those of ResNet-18. These models are MobileNetV2 (Sandler et al., 2018), MobileNetV3 (Howard et al., 2019), EfficientNet-(B0, B1, B2, B3) (Tan & Le, 2019), SqueezeNet1-0 (Iandola et al., 2016), and Densenet121 (Huang, Liu & Weinberger, 2016). For this purpose, we trained all these models using FP-MHI data for 35 epochs with the SGD optimizer in the same way that we trained ResNet-18, setting the initial learning rate to 1e−3 and reducing it by 1/10 every 15 epochs. Table 8 contains a comparison of these trained efficient models and ResNet-18. This comparison is based on several metrics: the test accuracy, parameter size, training time on GPU, inference time on GPU and CPU, total memory usage, and million floating-point operations per second (MFLOPS) value.

According to the comparison results in Table 8, the ResNet-18 model with the RReLU activation function outperformed the other models, especially in terms of the test accuracy, with a success rate of 92.37%. This performance is one of the reasons that ResNet-18 is the preferred model in our proposed deep learning-based SLR systems. The training time is relatively low for ResNet-18, with 10.75 s per epoch on GPU, while SqueezeNet has a faster training time. However, the accuracy of ResNet-18 outperformed the model with these faster training times. ResNet-18 offers very competitive inference times compared to other models, with values of 1.82 ms on GPU and 6.95 ms on CPU. This provides a significant advantage for real-time applications. The total memory usage was also reasonable at 55.71 MB, indicating that ResNet-18 can run effectively even on resource-constrained systems. The models were also evaluated in terms of MFLOPS values. The MFLOPS value refers to the maximum number of floating-point operations a model can perform in one second and is often used to evaluate the computational intensity and efficiency of a model. ResNet-18’s MFLOPS value of 489.79 indicates that the model requires relatively modest computational resources while providing a high accuracy, making it suitable for resource-constrained devices. This analysis reveals that ResNet-18 is a good choice for our proposed SLR task in terms of both accuracy and performance. Hence, the use of ResNet-18 in our proposed SLR system plays a critical role in improving the overall performance and effectiveness of our system.

Following the evaluation of the ResNet-18 model, a comprehensive analysis of the processing times of each process in our proposed final SLR system, which we created using eight ResNet-18s, was performed on a sample video taken from the BosphorusSign22k-general dataset consisting of 42 frames (1,920 × 1,080 resolution, 1,400 ms, 30 FPS). For this sample video, the system was run 100 times and average values were obtained. In addition, some optimizations were made to improve the temporal performance of the system. These optimizations included changing the model_complexity parameter of MediaPipe, rendering pose visualizations at a lower resolution, and performing feature extraction on the GPU. The temporal performance of each process is shown in milliseconds in the diagram in Fig. 10.

The total processing time was measured as 2,700 ms, including all operations starting from the extraction of the pose data and ending with the SVM classification. This time reflects the integrated operation of all components of the system. Pose extraction took 2,080 ms, the linear interpolation of missing poses took 54 ms, pose visualization took 369 ms, the calculation of MHI features took 110 ms, temporal sampling to select 32 frames took 0.12 ms, normalization took 14 ms, feature extraction from the ResNet-18 models took 55 ms, and SVM classification took 13 ms.

Since pose data are extracted from each frame in the video separately, pose data can be extracted during the reading of the video from a source or during the display of the video. Therefore, in order to extract pose data from a 1,400 ms video, it is expected that the video has been read from start to finish. This means that we can ignore the video duration for this exposure extraction time. Thus, for a video of roughly 1,400 ms, we can say that the pose extraction time cost is 680 ms. To confirm this, we showed the sample video on screen from start to finish and extracted the pose data. We saw that the video playback took 2,142 ms at about 20 FPS. By the end of the video playback, all pose data had been extracted. Considering the cost of displaying the video on the screen, we can say that pose data were obtained with a delay of 742 ms.

Pose data extraction accounts for approximately 77% of the total time of our system. This process is the most time-consuming step, and any improvement in this step will increase the temporal performance of the system. For this purpose, we updated the default value of MediaPipe’s model_complexity parameter from 1 to 0 and performed inference through a less complex model. As a result, the pose extraction time decreased from 2,080 to 1,756 ms, a gain of 324 ms. Another optimization that we made is in the pose visualization step. By reducing the 1,920 × 1,080 resolution pose images we created in this step by one-fifth, i.e., to 384 × 216, we saved 185 ms in the pose visualization process and created the relevant images in 145 ms. These images, which we created with a lower resolution, also affected the MHI extraction and “resize to 112 × 112” processes in the next processing steps, resulting in a total gain of 288 ms. We also performed feature extraction from the trained ResNet-18 models on the CPU by default. When we performed this processing step on the GPU, we achieved a gain of 22 ms. With these optimizations, our proposed method achieved a gain of 643 ms and an extraction time of 2,057 ms for a 1,400 ms sample video. When we ignore the video duration, we can say that the necessary operations were completed in 657 ms. These optimizations are an important step for our system to be used more effectively in real-time applications.

Another important aspect of these optimizations is the impact on memory usage. In our standard solution, the system used about 1.7 GB of memory, whereas after the optimizations, the memory usage was reduced to about 1.2 GB. This is a reduction of about 30% and allows the system to run more efficiently, especially on resource-constrained devices. Less memory usage also allows our system to be integrated in more applications, enabling more efficient multitasking.

We also evaluated the proposed system, with optimizations for temporal efficiency, using the entire BosphorusSign22k-general test set of 949 videos. This gave us the total inference time, as well as the effect of temporal optimizations on the accuracy. Table 9 shows the inference times and accuracies obtained from the analysis on the test set, consisting of 2,623 s in total.

In Table 9, the performance of the proposed SLR system is evaluated using different MediaPipe model complexities and pose visualization resolutions. The results show that reducing the model complexity (from 1 to 0) reduces the total inference time by about 12.5%, while reducing the resolution reduces the time by about 12.2%. The impact of these optimizations on the accuracy is also indicated in the table. Reducing the model complexity, possibly because it misses some pose data, and reducing the pose visualization resolution, because some details are lost, resulted in decreases in accuracy of 0.21% and 2.32%, respectively. These represent an important trade-off between the system performance and accuracy. These results show that our optimizations for temporal efficiency yield significant improvements, with acceptable reductions in the accuracy.

Comparison with other studies

After obtaining the results of our experimental work, it is important to compare the findings with other studies in the literature. In this subsection, we compare the performance of our proposed method with that of other studies. The comparisons will include the results obtained with our proposed FP-MHI feature and the final method obtained by fusing all features.

Table 10 presents the comparison of our experimental work on the BosphorusSign22k-general dataset with similar studies in the literature. Our proposed FP-MHI feature stands out with an accuracy of 92.37% compared to the methods of Kindiroglu, Ozdemir & Akarun (2019) and Gündüz & Polat (2021). In particular, compared to the complex system based on models such as Inception 3D and LSTM-RNN using body, hand, face, optical flow, and pose data presented by Gündüz & Polat (2021), our relatively simpler ResNet-18 model, in which FP-MHI features are integrated, provided an accuracy increase of 4.02%. This result against a model like Inception-3D that evaluates both spatial and temporal features shows that even with simple models, a high performance can be achieved through proper feature engineering and efficient algorithm design. Furthermore, the integration of body, hand, and face features in our system contributed to a significant test accuracy of 96.94% on the BosphorusSign22k-general dataset. This shows that our proposed method provides a significant improvement over other methods in the existing literature on sign language recognition and can provide a new reference point for research in this area.

Table 11 contains a comparison of methods applied to the BosphorusSign22k dataset. This dataset has rich content for sign language recognition research and covers 744 different classes. The repetition of our high results on the 174-class BosphorusSign22k-general dataset on this dataset shows the effectiveness and scalability of our method. Looking at the results, the approach presented by Özdemir et al. (2020), which extracts both temporal and spatial features from raw video data using the MC3-18 model, shows that our FP-MHI feature provides a 7.25% higher test accuracy compared to the 86.10% accuracy obtained with the ResNet-18 model, according to the IDT feature used in the same study (Özdemir et al., 2020). These results once again emphasize the importance of correct feature selection, as well as the complexity of deep learning models, in sign language recognition. Gökçe et al. (2020) obtained the highest accuracy rate on the BosphorusSign22k dataset. This work involves the use of the MC3-18 model, a powerful pre-trained model that can handle temporal and spatial information, as well as the integration of various components such as the body, hands, and face. Following a similar approach to these studies, our proposed integrated method achieves a competitive result of 94.87% accuracy with ResNet-18, a simpler model than MC3-18, by combining body, hand, face, and finger features. This result shows that temporal features can be significantly captured by the three-channel MHI features extracted from each piece of pose data. Furthermore, the use of RGB data to train the models in the study by Gökçe et al. (2020) and the fact that the dataset was built on a flat background suggests that the system may be potentially sensitive to different background conditions. This could be an important limitation in real-world applications, given the diversity of backgrounds. Our proposed method is based entirely on pose data, which is expected to make our system more robust to background variations. In this context, the success of our pose estimation method is a determining factor for the overall performance of our system. When effective pose estimation is achieved, the sign language recognition system can accurately recognize and interpret sign language components regardless of background variations. This feature is critical for our proposed system to work reliably in various application scenarios.

Table 12 shows the results of the experiments performed on the LSA64 dataset of 64 classes to verify the applicability of our proposed method in different sign languages. This table focuses on the results of the experiments in which the dataset is divided into three different training and testing conditions. E1 and E2 show the signer-independent evaluations, while E3 shows the results when the training and test set are randomly determined. In E1, there is only one previous study. In this evaluation, where signers 5 and 10 out of 10 signers are separated as test data, the accuracy rate of 74.22% obtained by Marais et al. (2022) is increased to 94.68% with our proposed FP-MHI method and 98.43% with the Fusing+SVM method. These results may be a new starting point for signer-independent evaluations. In E2, where each of the 10 signers was used as a separate test, the experiments were repeated 10 times, and the average accuracies were obtained; the results obtained with our proposed FP-MHI feature and the Fusing+SVM method surpassed all other studies. Alyami, Luqman & Hammoudeh (2023)’s work on the classification of keypoints from the signer’s hands and face with the transformer model, which was one of the latest studies included in this analysis, falls short of the result obtained by classifying FP-MHIs, which are finger-based features, with ResNet-18. For E3, the 100% accuracy rate obtained by Bohacek & Hruz (2022) is equivalent to our proposed methods. In this study, pose data are used in a similar way to our study. This proves the effectiveness of pose data in sign language recognition. For E3, our proposed methodology achieves a 98.90% accuracy with FP-MHIs and a 100% accuracy with Fusing+SVM. Overall, this detailed comparison shows that our proposed methodologies perform competitively and even outperform existing work in the field of sign language recognition, in both the signer-independent and randomized split test conditions.

Another dataset that we used to verify the applicability of our proposed method to different sign languages is the GSL dataset, which contains 310 different Greek Sign Language words. Table 9 shows the studies on the GSL dataset. The same author names refer to the results of different methods from the same article. In the experiments performed on the GSL dataset, the proposed FP-MHI method provides a similar accuracy, especially compared to some recent studies. In particular, it provides results close to the results of Adaloglou et al. (2020). However, our proposed Fusion + SVM method stands out with a higher accuracy rate than almost all studies in the literature. In particular, it is very close to the highest accuracy rate obtained by Selvaraj et al. (2021). In conclusion, the experiments performed on the GSL dataset show that our proposed methods are effective and competitive with other approaches in the literature.

In summary, when the data from Tables 10–13 are analyzed, we can see that our proposed methods provide highly competitive and, in some cases, superior results compared to existing methods in the literature on a wide range of different datasets. In particular, the FP-MHI feature stands out with its performance on various datasets. On some datasets, this feature made it possible to obtain similar or even better results than existing methods in the literature. This impressive performance of the FP-MHI feature shows how valuable and effective this feature is in sign language classification. The achieved accuracy rates of our method, which incorporates the fusion of all features and utilizes an SVM for classification, are comparable to or surpass the highest reported outcomes in the existing literature. This demonstrates that the feature extraction and fusion approach employed in this study is highly efficacious in addressing the sign language classification problem. Moreover, the demonstrated consistency of the experimental results indicates that the employed methodologies possess reliability and robustness across diverse datasets.