Recognition of sports and daily activities through deep learning and convolutional block attention

Data denoising

The raw data acquired from sensors carried measurement noise from system limitations or unexpected movements during the experiments. This noisy signal interferes with the valuable information embedded in the data. Consequently, it was imperative to mitigate the impact of noise to extract meaningful information for subsequent processing. Various filtering methods are commonly employed to handle this issue, including the mean filter, low-pass filter, Wavelet filter, and Gaussian filter (Rong et al., 2007; Mostayed et al., 2008). In this study, we opted for an average smoothing filter to denoise the signal, applying the filter across all three dimensions of the accelerometer and gyroscope sensors.

Data normalization

Subsequently, the raw data from sensors undergo normalization, ensuring their values fall within the range of 0 to 1, as shown in Eq. (1). This step addresses challenges in model learning by harmonizing the data values to a uniform range, facilitating quicker convergence during gradient descents. (1)${\displaystyle X_i^{norm}=\frac{X_i-x_i^{min}}{x_i^{max}-x_i^{min}},i=1,2,3,\dots }$ where $X_i^{\mathrm{norm}}$ represented the normalized data, n represented the number of channels, $x_i^{max}$ and $x_i^{min}$ are the maximum and minimum values of the ith channel, respectively.

Data segmentation

As wearable sensors produce a large amount of signal data, it is impractical to simultaneously input all of this data into the HAR model. Therefore, the data must be divided into segments using a sliding window approach before being fed into the model. Splitting data into windows is a widely employed technique in HAR to separate data for analysis. It is commonly used to detect recurring activities such as walking, jogging, and inactive, including standing, sitting, and lying down (Banos et al., 2014). The continuous sensor signals are split into fixed-length windows. Consecutive windows overlap partially to increase the number of available training examples and avoid missing transitions between activities. Figure 3 demonstrates the windowing process.

The segmented sample data, obtained using a sliding window of size N, has a dimension of K × N. The given sample is represented by the symbol W_t. (2)${\displaystyle W_t=\left[a_t^1{a}_t^2...a_t^K\right]\in {\mathbb{R}}^{K\times N}}$

The column vector $a_t^k={\left(a_t^1,a_t^2,\dots ,a_{t_N}^k\right)}^T$ represents the signal data of sensor k at window time t. T denotes the transpose operator, K represents the number of sensors, and N indicates the length of the sliding window. To exploit the relationships between windows and facilitate the training process, the window data is divided into sequences of windows. (3)${\displaystyle S=\left\{\left(W_1,y_1\right),\left(W_2,y_2\right),\dots ,\left(W_T,y_T\right)\right\}}$

T is the length of the window series, and y_t represents the activity label of the associated window W. The label of a window that contains several activity classes will be determined by selecting the most often occurring sample activity.

Convolution block

CNNs rely on a standard set of components. They often apply supervised learning methods. Typically, CNN neurons connect to every neuron in the next layers. An activation function transforms the neuron’s inputs into outputs. Two key factors impact the activation function’s efficacy. One is sparsity—the fraction of zero activations. Another is ensuring sufficient gradient flow as the network depth increases. CNNs frequently use pooling to reduce dimensionality. Two common techniques are taking the maximum (max-pooling) or average (average-pooling) value over input regions. Max-pooling and average-pooling simplify the representations by reducing the number of parameters within activation maps. At the same time, they preserve the most salient information.

This work involves employing convolutional blocks (ConvB) for identifying essential characteristics in raw sensor data. The diagram in Fig. 4 demonstrates that ConvB comprises six layers of 1D-convolutional (Conv1D) and batch normalization (BN). The structure is made up of three layers of Conv1D-BN. Conv1D employs multiple trainable convolutional kernels to capture diverse attributes, with each kernel producing a distinct feature map. To optimize the training process’s efficiency and speed, we opted to incorporate the BN layer.

BiGRU block

At the core of our proposed CNN-BiGRU-CBAM model lies the BiGRU. This crucial component captures temporal correlations and context from the features extracted by the convolutional layers. Unlike conventional RNNs, BiGRU excels in processing sequential input by maintaining an internal state and employing gating mechanisms to control information flow.

In a standard GRU, the hidden state h_t at time t isdetermined by combining the outputs of the update gate z_t, reset gate r_t, current input x_t, and the previously hidden state h_t−1. The update gate z_t determines the amount of information to retain from the previous hidden state, while the reset gate r_t determines the degree to which the previous hidden state should be ignored. Figure 5A illustrates the structure of a GRU cell and the mathematical equations managing its operation as follows: (4)${\displaystyle z_t=\sigma \left(W_z{x}_t\oplus U_z{h}_{t-1}\right)}$ (5)${\displaystyle r_t=\sigma \left(W_r{x}_t\oplus U_r{h}_{t-1}\right)}$ (6)${\displaystyle g_t=tanh\left(W_g{x}_t\oplus U_g\left(r_t\otimes h_{t-1}\right)\right)}$ (7)${\displaystyle h_t=\left(\left(1-z_t\right)\otimes h_{t-1}\right)\oplus \left(z_t\otimes g_t\right)}$

Here, σ represents a sigmoid function, ⊕ denotes an element-wise addition, and ⊗ denotes an element-wise multiplication.

To augment the capacity of the GRU in apprehending historical and forthcoming context, we integrate a BiGRU into our framework. Illustrated in Fig. 5B, a BiGRU comprises two distinct GRU strata: a forward GRU, which handles the input sequence from the initial timestep to the concluding one, and a backward GRU, which inversely manages the sequence. Subsequently, the outputs of both GRU layers are combined at each timestep to generate the ultimate BiGRU output, encompassing insights from both antecedent and subsequent contexts.

Within our CNN-BiGRU-CBAM architecture, the BiGRU segment receives the spatial characteristics derived from the convolutional strata as its input. Independently, the forward and backward GRU strata analyze these characteristics, grasping the temporal connections and context in opposing directions. By amalgamating the results of both GRU layers, the BiGRU component furnishes a more exhaustive comprehension of the temporal correlations among the extracted features. Consequently, this enhancement empowers the model with improved capabilities to discern and categorize diverse sports and daily endeavors.

Incorporating a BiGRU component in our model presents numerous benefits compared to a single-direction GRU. By encompassing historical and forthcoming contexts, the BiGRU can apprehend richer temporal structures and interrelations, thereby enhancing activity recognition performance. Furthermore, the bidirectional processing aids the model in effectively managing intricate and diverse-length sequences, which are prevalent in sensor-based HAR tasks.

CBAM block

The CBAM proposal was put forth in Woo et al. (2018) as an attention mechanism to improve performance by amplifying significant channels and crucial sections of intermediate features. CBAM comprises two sub-modules: the channel and spatial attention modules, as depicted in Fig. 6. These modules are intended for application post-convolutional layers, as their names indicate. CBAM capitalizes on features’ spatial and cross-channel connections by progressively integrating channel and spatial attention. To be more specific, it accentuates beneficial channels and reinforces informative local regions.

Figure 6 demonstrates the CBAM attention mechanism, incorporating channel and spatial attention modules. The initial passage of data involves the channel attention module (CAM), illustrated in Fig. 7. CAM employs adaptive learning to assess the importance of each channel, facilitating more precise selection and utilization of information from diverse channels. Subsequently, the data undergoes processing in the spatial attention module, depicted in Fig. 8. This module enhances the network’s focus on various spatial positions, enabling a deeper understanding and effectively exploiting significant characteristics in distinct locations. These modules can augment the network’s ability to perceive and withstand challenges (Agac & Durmaz Incel, 2023).

Figure 7 illustrates the structure of the CAM module. Within the CAM module, both maximum pooling, enhancing sharpness, and average pooling, providing a smoothing effect, are applied in the spatial dimension of the input feature. Subsequently, the input feature is subjected to a multilayer perceptron (MLP) transformation utilizing a reduction ratio (r), and eventually, a sigmoid activation function is used. The reduction ratio is crucial in determining the extent of dimensionality reduction. It helps maintain a balance between computational effectiveness and attention accuracy by using a shared MLP component in the channel attention process. A lower reduction ratio can amplify the channel attention mechanism’s information expression but escalates computational complexity. Conversely, a higher reduction ratio trims computational complexity but may constrain the channel attention mechanism’s expressive capability. Achieving an optimal trade-off between attention performance and computing efficiency necessitates fine-tuning the reduction ratio based on the specific application.

The spatial attention module (SAM) system consists of three consecutive steps, as illustrated in Fig. 8. Initially, two tensors are formed by implementing maximum and average pooling operations on the channels of the input feature F. The two tensors are merged and inputted into the convolutional layer with a defined kernel size to generate a feature map with a single channel. Subsequently, the output undergoes processing in the sigmoid activation layer to obtain the final spatial attention mask. Finally, each feature map in X undergoes element-wise multiplication with the constructed spatial attention mask.

Impact of the convolution block

To explore the impact of the convolutional block on model performance, we conducted an ablation study using the UCI-DSA dataset. This involved creating a baseline model by excluding the convolutional block from the original CNN-BiGRU-CBAM architecture while retaining the BiGRU and CBAM blocks. The baseline model directly feeds raw sensor data into the BiGRU block without any spatial feature extraction.

Table 6 presents the findings of the ablation investigation. The model without the convolutional blocks exhibits inferior recognition performance compared to the complete CNN-BiGRU-CBAM model. Removing the convolutional blocks decreases the F1-score from 99.51% to 98.65%, underscoring the importance of spatial feature extraction for accurate activity identification.

Impact of the BiGRU block

The BiGRU component within our CNN-BiGRU-CBAM framework plays a crucial role in capturing temporal connections and context from the features extracted by the CNN layers. GRUs are a specific type of RNN renowned for processing sequential data by maintaining internal states that retain and recall information from previous time steps.

Our model employs a BiGRU, comprising two GRU layers that process the input sequence in both forward and backward directions. The forward GRU sequentially analyzes the input sequence from the initial to the final time step, while the backward GRU examines the sequence in reverse, from the final to the initial time step. This bidirectional approach allows the BiGRU to capture context from past and future time steps, providing a comprehensive understanding of temporal relationships between the extracted features.

The BiGRU component receives the CNN layers’ output as input, representing spatial characteristics derived from sensor data. The input sequence undergoes separate processing by the forward and backward GRU layers, with their outputs combined at each time step to form the final BiGRU output. This output encapsulates sequential connections and contextual information from the extracted features, facilitating a deeper comprehension of the relationships between different actions and enhancing overall identification accuracy.

To evaluate the BiGRU block’s effectiveness in capturing temporal relationships, we conducted an ablation experiment on our proposed CNN-BiGRU-CBAM model. The baseline architecture for this experiment was a modified version of CNN-BiGRU-CBAM lacking a BiGRU block. The results in Table 7 confidently indicate that the CNN-BiGRU-CBAM model, incorporating the BiGRU block, outperforms the baseline model without it. Specifically, it achieves an increase of approximately 1.27% in F1-score on the UCI-DSA dataset.

The BiGRU block’s bidirectional processing capability is advantageous for activity recognition tasks as it integrates information from past and future time steps, which is crucial for distinguishing between various activities. By capturing relevant context, the BiGRU block enhances the model’s ability to accurately identify and categorize different sports and daily activities using sensor data.

Impact of the CBAM block

The CBAM block, a novel addition to our proposed CNN-BiGRU-CBAM model, is designed to enhance the model’s ability to focus on crucial features for activity detection. This is achieved by implementing attention mechanisms in both the channel and spatial dimensions, a unique approach in the field of computer vision and machine learning. This block consists of two consecutive submodules: the channel and spatial attention modules. The former exploits inter-channel interactions to generate a channel attention map, emphasizing informative channels, while the latter utilizes inter-spatial interactions to develop a spatial attention map, highlighting relevant spatial locations.

Positioned after the BiGRU block in our model, the CBAM block refines spatial–temporal feature representations before reaching the final classification layer. By employing attention mechanisms, the CBAM block enables the model to prioritize essential areas in feature maps while filtering out noise and irrelevant data. This attention-based feature refinement is believed to enhance the model’s ability to detect activities effectively across various body postures, a significant advancement in the field of activity detection.

To evaluate the impact of the CBAM block, we conducted an ablation study, removing the CBAM block from the CNN-BiGRU-CBAM model while retaining the convolutional and BiGRU modules. The performance of the ablated model was assessed on all three benchmark datasets and compared to the complete CNN-BiGRU-CBAM model.

The results of the ablation study, presented in Table 8, demonstrate that the CNN-BiGRU-CBAM model incorporating the CBAM block achieves superior recognition performance compared to the ablated model without the CBAM block. Integration of the CBAM block leads to a 0.36% improvement in the F1-score for the UCI-DSA dataset. This enhancement can be attributed to the attention mechanism’s ability to highlight important features and reduce interference, thereby enhancing the model’s ability to discriminate between different elements.

In summary, the CNN-BiGRU-CBAM model employs convolutional modules, a BiGRU component, and a CBAM module to effectively extract the most relevant features from the data captured by wearable sensors.

Memory consumption

During the inference process on test dataset batches, we evaluated the memory usage of the CNN-BiGRU-CBAM architecture compared to the baseline CNN, LSTM, BiLSTM, GRU, and BiGRU architectures using CUDA profiling tools. The results presented in Table 9 reveal that our proposed model requires an average of 3,914,472 bytes of memory. This memory stores encoder feature maps, temporal hidden states, classifier weights, and outputs. In contrast, the basic CNN consumes an average of 6,147,880 bytes when processing shorter feature sequences. Conversely, the standalone BiGRU architecture occupies an average of 2,091,160 bytes without employing convolutions to represent spatial hierarchies. The CNN-BiGRU-CBAM model strikes a balance by enhancing accuracy without substantially increasing memory usage.

Prediction time

To evaluate efficiency, we analyze the complexity of the models based on their mean prediction time. This metric is determined by feeding a set of samples from the test data into the HAR models and calculating the average duration required to process one window. The results of this analysis are presented in Table 9, showcasing the average prediction time in milliseconds for the deep learning models operating on the UCI-DSA dataset. BiGRU exhibits the most prolonged prediction latency among these models, with a duration of 0.416 ms. It is worth noting that BiGRUs typically require a significantly longer time during training than CNN-based models, including our proposed CNN-BiGRU-CBAM. This is because convolutions can be executed concurrently, only needing a few adjacent values to compute the output of a single kernel. However, with BiGRU, a substantial portion of computations must be performed sequentially, as outputs depend on prior outputs. The mean prediction time of the CNN-BiGRU-CBAM model is 0.425 ms, comparable to that of the BiGRU and BiLSTM models.

Trainable parameters

In addition to memory usage and average prediction time, we consider the number of trainable parameters to indicate model complexity. These parameters represent the weights that are adjusted during the model training process, and a higher count suggests the model’s capacity to convey more intricate data. However, if the data lacks complexity, it also increases the risk of overfitting.

The count of trainable parameters for the deep learning models (CNN, LSTM, BiLSTM, GRU, BiGRU, and the proposed CNN-BiGRU-CBAM) utilized in this study is meticulously presented in Table 9. These values were obtained from the model summary using the benchmark UCI-DSA dataset, ensuring the accuracy and reliability of our findings. The results align with the intuitive understanding of the complexity differences among these models. Among the fundamental deep learning models, the GRU is the simplest, with 86,163 parameters, while the CNN is the most complex, with 509,011 parameters. The proposed CNN-BiGRU-CBAM model contains a total of 312,698 trainable parameters. This figure is notably higher than those of other deep learning models but remains lower than that of the CNN model.