Attention-enhanced gated recurrent unit for action recognition in tennis

Model architecture

Figure 2 describes the model architecture proposed in our study. Each video is cropped into 16 frames for action recognition. These frames are then embedded using pre-trained encoders (Szegedy et al., 2016) to create corresponding vectors of size 15×2048. Three different pre-trained models, including InceptionV3 (Szegedy et al., 2016), DenseNet (Huang et al., 2017), and EfficientNetB5 (Tan & Le, 2019), were used for featurization. These vectors then enter two gated recurrent unit (GRU) layers of hidden dimension sizes of 256, respectively. The output features are learned in the attention layer to selectively learn essential characteristics. Finally, the attended features are transferred to the fully connected (FC) layer to return final outcomes. The model was trained over 100 epochs with a batch of 32 samples. The learning rate was fixed at 0.0001, and the training time per epoch is estimated to be about 0.2 s.

Attention layer

An attention layer constitutes a pivotal component within DL architectures, prominently featured in natural language processing and computer vision domains (Vaswani et al., 2017). This layer facilitates the selective weighting of input data elements, enabling models to emphasize pertinent information while diminishing the significance of extraneous details. This mechanism closely mirrors human attention mechanisms, rendering it particularly efficacious in tasks such as machine translation, sentiment analysis, and image captioning. Adding attention layers to models helps them understand context better and capture long-range dependencies. This is a key part of getting better results in many AI tasks.

In this work, the attention layer was designed to detect which source objects are associated with the next target object and assign suitable attention weights when computing the context vector c_j. Given embedded source representation H = {h₁, h₂, h₃, …, h_n} and the previous decoded state s_j−1, c_j can be simply expressed as: (1)${\displaystyle c_j=Attention\left(H,s_{j-1}\right).}$

Initially, the attention weight α_j,i is computed to estimate the degree of association of a source object x_i when predicting the target object y_j via a feed-forward network: (2)${\displaystyle {\alpha }_{j,i}=\frac{exp\left(e_{j,i}\right)}{\sum _{k}exp\left(e_{j,k}\right)},}$ where the association score e_j,i is derived from Bahdanau, Cho & Bengio (2014) as: (3)${\displaystyle e_{j,i}=v_{\alpha }^T\odot tanh\left(W_{\alpha }\cdot s_{j-1}+U_{\alpha }\cdot h_i\right).}$

Basically, the greater attention weight α_j,i refers to the higher importance of object x_i for the next object prediction. Hence, Attention computation creates c_j by directly assigning weights for the source representations H with their corresponding attention weights ${\alpha }_{i-1}^n$: (4)${\displaystyle c_j=\sum _i{\alpha }_{j,i}\cdot h_i.}$

Image embedding

Image embedding in DL refers to the process of transforming high-dimensional image data into a lower-dimensional representation that retains essential information about the image’s content and characteristics. This lower-dimensional representation, known as an image embedding or feature vector, is designed to capture semantically meaningful features and patterns within the image. Image embeddings are valuable because they facilitate various computer vision tasks, such as image retrieval, object detection, and image similarity analysis. Deep learning models, especially CNNs, are commonly employed to generate image embeddings by extracting hierarchical and abstract features from the input image. The resulting image embeddings are not only more compact but also contain valuable semantic information, making them suitable for applications where efficient and meaningful image representations are required.

InceptionV3 is a pre-trained CNN model developed by Google (Szegedy et al., 2016). It is now known as one of the effective inception modules that facilitate multi-scale feature extraction. These modules enable the network to effectively recognize objects and patterns of different sizes within images. InceptionV3 also incorporates optimizations like batch normalization and factorized convolutional layers to enhance training stability and reduce computational complexity. Trained on large datasets like ImageNet (Deng et al., 2009), InceptionV3 is widely used in computer vision for tasks such as image recognition and object detection due to its efficiency and robust performance.

DenseNet is another pre-trained CNN model designed for computer vision tasks (Huang et al., 2017). Its unique feature is dense connectivity, where each layer directly connects to every other layer, enabling efficient feature reuse and better gradient flow. DenseNet models, like DenseNet-121 and DenseNet-169, have gained popularity for their ability to learn rich features effectively, making them a top choice for tasks like image classification, object detection, and segmentation due to their superior performance and parameter efficiency.

EfficientNet, a pioneering deep learning architecture developed by Google, has been known for its exceptional computational efficiency and concurrent model performance improvement (Tan & Le, 2019). This innovation relies on compound scaling, simultaneously optimizing network depth, width, and resolution, resulting in notable advancements in various computer vision tasks. Its capacity to strike a delicate balance between model complexity and predictive accuracy has made EfficientNet a preferred choice for machine learning practitioners, offering quicker training and deployment while maintaining competitive state-of-the-art performance. This scalability and adaptability have profound implications for resource-efficient and versatile deep learning solutions across diverse applications in artificial intelligence.