Deep learning-based classification with improved time resolution for physical activities of children

Methods overview

Developing a physical activity classification algorithm requires the acquisition of actual acceleration data from body movements of target-age children. This study formed a research plan for this data acquisition including subject recruitment, a series of physical activity protocols, and the preparation of equipment for recording physical activity data.

Acquired data were processed into an appropriate form for generating datasets, which were then used to develop an algorithm classifying the physical activities of the children. A deep machine learning method was used to ensure high algorithm performance. This deep machine learning algorithm used in conjunction with the datasets proved capable of classifying the physical activities performed children.

Preparation of data acquisition

A total of 136 participants (86 boys and 50 girls) took part in this experiment from one school and various sports clubs. From these 136 participants, we obtained valid data for 115 subjects consisting of 75 boys and 40 girls. The data from 21 subjects were dropped owing to pre-existing physical or mental illnesses/conditions, and in some instances, the data was invalid owing to device malfunctions or unexpected noise. The ages of participants for whom the data were valid were distributed between 8.5 and 12.5 years with a mean of 10.5, and a standard deviation of 1.1.

An experimental protocol explaining the types of movements to be conducted, as well as when and how long they should last, was designed to obtain the physical activity data generated by the body movements. Informed consent was obtained from the participants as well as from their parents. The consent form explained the contents of the experiment and the protocol. The experimental plan and consent form were approved by the internal review board of the Catholic University of Korea, Seoul ST. Mary’s Hospital (KIRB-00472_9-002).

Participants fulfilled protocols composed of various movements, including walking, running, and moving on stairs. Each individual wore an accelerometer module on their waist between the center of the belly and right pelvic, along the waistline of the pants (standard positioning of a pedometer) as they completed the protocol. The wearable data-recording module used in this study was designed as an accelerometer system to collect acceleration signals generated by body movements and fabricated from a microcontroller unit, three-axis acceleration sensor, memory chip, power unit, etc. Figure 1 shows the appearance of the devised system. The accelerometer module had a single three-axis accelerometer and no other sensors, thereby minimizing the use of sensor signals and power while maximizing the portability and convenience. The module accelerometer range was set to 4 × gravity in both positive and negative directions (±4 g). The module is 50 mm ×30 mm ×15 mm in dimension, weighs 21 g, and has a case with a clip to fasten it to the wearing point of pants.

Data acquisition procedure

Participants followed the physical activity protocol composed of various motions including sitting down on and standing up from a chair, jumping in place, walking, running, ascending and descending stairs, and jumping rope. A table sheet with a summarized protocol listing selected physical activities was used during the experiment and the physical activity classification algorithm development. Table 1 shows the protocol of selected activities. The net protocol conduction time, excluding resting, was 22 min. Although subjects took breaks between protocol activities to catch their breath and recover their stamina, break times were not marked in the table sheet because each subject had a different recovery condition and the length of break periods varied widely. Recovery break times were based on the criterion of heart rate, such that subjects were considered to have recovered when their heart rates approached a resting level. The usual break time ranged between 2 and 5 min according to the activities.

Table 1 shows three activity groups: Group A lists a still state and intermittent movements, group B lists basic and highly frequent steady-state movements (i.e., walking and running), and group C lists relatively infrequent steady-state movements (i.e., moving on stairs and jumping rope).

As the order of physical activities could influence movement patterns, the effect of ordering was considered before subjects performed the protocol. For example, successive vigorous activities could make create differences in movement relative to other activity sequences even with sufficient resting times. To circumvent this ordering effect, subjects were evenly divided into six shuffled group orderings: A-B-C, A-C-B, B-A-C, B-C-A, C-A-B, and C-B-A. Group B activities (walking and running) were repeated at slow and fast speeds determined entirely by the participants.

Data preprocessing and learning for classification

A. Data preprocessing

1. Data collection and filtering

   While the participants performed physical activities according to the above protocol, the accelerometers they wore measured acceleration signals that were then digitized by the microcontroller unit and stored in a mounted memory chip. Data sampling was performed at a frequency of 45.4 Hz. Stored data files containing all activity data segments for an ordered protocol (as in Fig. 2A) were then downloaded onto a PC for data processing. A high-pass frequency domain filter (over 0.5 Hz) was used to eliminate the bias signal from data.

2. Data samples and augmentation

   The filtered data were divided into protocol activities and cut to constant time windows with widths of 128 data points, corresponding to ∼2.8 s. At this stage, data augmentation with overlaps and rotations allowed us to enrich and diversify our data samples. Adjacent windows shared half their data with an overlap of 50%. In addition, the data was rotated at random degrees within ±10 for yawing, ±15 for pitching, and ±20 for rolling. This was used to simulate the rotated state of equipped accelerometer. Figure 2B depicts the windowing and cutting data processes.

3. Data preparation for learning

   The augmented window-sized data samples from all participant data (∼184,000 samples as listed in Table 1) were sorted by activity to form ten dataset types, one for each activity: slow walking (WS), fast walking (WF), slow running (RS), fast running (RF), walking up the stairs (SU), walking down the stairs (SD), jumping rope (JR), standing up (ST), sitting down (SI), and keeping still without activity (NA). Figure 2C shows the assignment of window-sized data samples to their corresponding datasets. The dataset bundles for the ten activities were then fed into the CNN training process.

B. Convolutional feature extraction

Figure 3 shows the CNN structure and describes the training procedure. The overall network architecture was composed of an input stage, a feature extraction stage with three convolutional layer blocks, and an output provision classification stage. The input stage received the preprocessed dataset. The feature extraction stage followed a triple ternary convolution block structure including convolution, pooling, and activation functions. Each convolution block is expressed in Fig. 3 with a different color. The last classification stage consisted of a fully connected layer, dropout, and softmax operator. The final network outputs came from the softmax operation results.

1. Input layer

   In a CNN, a layer can be represented by the general expression function y = f(x) where both the input x and output y are tensors. The network input is a Height(H) × Width(W) × Dimension(D) sized tensor data. In this study, the input was the time series accelerometer data where H = window size, W = single dimension, and D = number of sensor axes. As shown in Fig. 3, this input size H × W × D was equal to 128 × 1 × 3.

2. Convolution layer

   The convolution layer computed the input x convolution using filters f. The element expressions can be represented as: (1)${\displaystyle \mathbf{x}\in {\mathbb{R}}^{H\times W\times D},\mathit{f}\in {\mathbb{R}}^{H^{{}^{\prime }}\times W^{{}^{\prime }}\times D\times D^{{}^{\prime \prime }}},\mathbf{y}\in {\mathbb{R}}^{H^{{}^{\prime \prime }}\times W^{{}^{\prime \prime }}\times D^{{}^{\prime \prime }}}.}$ The output of this convolution layer was the three-dimensional convolution result, as represented below in Eq. (2) where i, j are spatial subscripts and d represents depth. (2)${\displaystyle y_{i″j″d″}=b_{d″}+\sum _{i^{\prime }=1}^{H^{\prime }}\sum _{j^{\prime }=1}^{W^{\prime }}\sum _{d^{\prime }=1}^D{f}_{i^{\prime }{j}^{\prime }d}\times x_{i″+i^{\prime }-1,j″+j^{\prime }-1,d^{\prime },d″}.}$

3. Pooling layer

   Pooling layers are used to identify the most important features and compresses layer sizes by subsampling. In this study, a max pooling method (as in Eq. (3)) compared consecutive convolutional result values in H′ × W′ sized areas and the maximum value survived to go through to next layer. The comparison area (H′ × W′ = 2 × 1) was exclusive, decreasing the spatial resolution by the factor of area size. (3)${\displaystyle y_{i″j″d}=\underset{1\le i^{\prime }\le H^{\prime },1\le j^{\prime }\le W^{\prime }}{max}{x}_{i″+i^{\prime }-1,j″+j^{\prime }-1,d}}$

4. Activation function

   An activation function followed the max pooling step in the convolution block. A rectified linear unit (ReLU), as expressed in Eq. (4), was selected for this function to mitigate the vanishing gradient problem and reduce learning time (Glorot, Bordes & Bengio, 2011). Any negative output values remaining after the max pooling layer were set to zero. (4)${\displaystyle y_{ijd}=max\left\{0,x_{ijd}\right\}.}$

5. Repeated convolution blocks

   Three convolution blocks were used, combining the steps of convolution, max pooling, and ReLU. Figure 3 shows the ternary convolution block structure, the filter sizes of which were 7 × 1 × 3 × 72, 6 × 1 × 1 × 144, and 5 × 1 × 1 × 108, respectively.

C. Classification for output

1. Fully connected layer

   The ReLU results of each layer, as with all layer outputs, were connected to the weighted nodes of the next layer. This study featured 256 empirically set nodes.

2. Dropout layer

   A dropout layer was imported to prevent overfitting, which may occur if a node in one layer excessively influences the subsequent layer. The dropout step dropped randomly selected nodes in fully connected layers with a set turn-on rate. Regularizing dropouts thinned the ongoing training network and created an ensemble effect within the thinned network (Hinton et al., 2012; Srivastava et al., 2014). This step could be performed in every fully connected layer except the final one for output. In this study case, only a single dropout layer was inserted in the penultimate fully connected layer using a rate of 0.5.

3. Softmax layer

   The softmax function was applied across feature channels to consolidate network results. This normalization method was applied to make that the sum of output probability distributions 1, as shown in Eq. (5). (5)${\displaystyle y_{ijk}=\frac{e^{x_{ijk}}}{\sum _{t=1}^D{e}^{x_{ijk}}}.}$

4. Classification losses

   The categorical loss function l(x, c) calculates the difference between a prediction x and ground truth c. The classification error l is zero if the predicted class with the largest score is same to ground truth, otherwise the error is 1.

D. Training and validation strategy

The stochastic gradient descent (SGD) method was used for backpropagation during the training process. The gradients for each parameter were averaged over the training cases in each batch. The learning rate started at 0.0003 and decreased by a log scale function with a factor of 0.5 for every hundred epochs. The momentum and weight decay parameters were set to 0.9 and 0.0005, respectively, and the batch size (set by experimental trials) was 128. It was determined the training epoch was optimized just before overfitting. The critical point was determined to be when the slope of the log loss curve changed from negative to positive, although the classification error continued decreasing past this point. Epoch points (such as the global minimum of the error function) were considered to be optimal points. Finally, the trained network was validated using the 10-fold cross-validation method. Ten groups of data were established such that nine groups were used for training and the remaining group was used for validation. Each group was designed to participate in validation in turn. The performance was measured using the mean of the validation results for each group.