Dense symmetric temporal alignment learning for human pose estimation

Image-based human pose estimation

Human pose estimation has gained significant attention in recent years. For example, Wei et al. (2016) proposed a sequential structure model based on CNN, which generated more accurate results through iterative optimization of each module in the training process. Fang et al. (2017) designed a symmetric spatial feature encoding network to capture high-quality pose information from inaccurate bounding boxes. Sun et al. (2019) built a high-to-low resolution model to maintain multi-resolution mapping features through feature extraction, which fuses various resolution representations to improve estimation accuracy. Moon, Chang & Lee (2019) proposed a fully learning-based camera distance-aware top-down general framework compatible with most existing human body detection and estimation modules. Kreiss, Bertoni & Alahi (2019) introduced a bottom-up method for multi-person 2D pose estimation, which leveraging part intensity fields (PIF) for body part localization and part association fields (PAF) for pose formation. Maji et al. (2022) developed You Only Look Once (YOLO)-pose, a heatmap-free method for 2D pose estimation, optimizing the object keypoint similarity metric end-to-end and detecting poses in a single forward pass.

Video-based human pose estimation

Nevertheless, utilizing image-based approaches to video may leads to reduced performance, due to motion blur, occlusion, and similar challenges. For solving these problems, Song et al. (2017) extracted relevant movement features from the temporal domain by calculating the dense optical flow features between adjacent frames. Luo et al. (2018) designed an end-to-end architecture that employed a loopy spatio-temporal graph to encode the consistency of human poses in videos. Bertasius et al. (2019) proposed a pose warper network based on a space-time distortion mechanism, which improves the label propagation between frames and benefits the training from sparsely labeled videos. Fang et al. (2023) combined symmetric integral keypoint regression and pose aware identity embedding, enhanced by PGPG and knowledge distillation, to accurately localize human keypoints. Peng, Zheng & Chen (2024) introduced a dual-augmentor framework that enhances pose estimation generalization through meta-optimization and differential strategies. Mehraban, Adeli & Taati (2024) built on AGFormer blocks that combine transformer and GCNFormer streams to enhance 3D structure learning through adaptive fusion, offering four variants for speed-accuracy trade-offs. Li et al. (2025) designed a unified MLP-GCN architecture for 3D pose estimation, which efficiently modeling spatial and temporal dynamics with minimal computational cost. Li et al. (2024) proposed hourglass tokenizer (HoT) for efficient 3D pose estimation, including a token pruning cluster to reduce redundancy and a token recovering attention to restore full-length temporal resolution.

Global information alignment block

Ideally, incorporating features from similar positions of adjacent frames will obtain the optimal representation of the human pose to improve estimation accuracy. However, the rapid movements of persons or cameras cause significant spatial shifts of person between each frame. Existing algorithms direct fuse features from neighboring frames, which inevitably introduce noise information that is unrelated with the current frame. We instead propose the global information alignment block (GAB) to obtain a coarse preliminary alignment of the supporting frame. The GAB computes the spatial alignment parameters of the global affine transform to achieve coarse preliminary alignments between neighboring frames. It is worth noting that, unlike existing methods that perform feature alignment using several deformable convolution layers, we employ a symmetric architecture that repeatedly employs deformable convolutions to achieve more adequate feature alignment.

As we know, the simple linear mapping or regular convolution network can be empolyed to capture spatial features, but it fails to extract satisfactory information from the complex and variable human poses. In contrast, the deformable convolution network able to adaptively transform the sampling point of the convolution kernel to selectively encode information from various spatial locations. Motivated by this, we design GAB based on deformable convolution network to globally align the supporting frames $I_s=⟨I_{t-2},I_{t-1},I_{t-1},I_{t+2}⟩$ with the current frame $I_t$ at coarse level.

To facilitate the representation of this operation process, we take the alignment stage of neighboring frame $I_{t-1}$ as an example.

We first utilize the high-resolution network as the backbone, the state-of-the-art image-based human pose estimation method, to extract the features from the current frame and the neighboring frames to obtain the feature vector $F_t$ and $F_{t-1}$, respectively. The HRNet is pre-trained on the human pose estimation dataset (COCO dataset) to obtain high-resolution human features. The feature vector $F_t$ and $F_{t-1}$ feed our GAM to compute global rearrangement parameters to obtain a preliminary alignment of the supporting frame feature with the current frame feature.

Specifically, the process stage of GAB is as follows:

Given $F_t$ and $F_{t-1}$, the sampling offsets of each pixel in the convolutional kernel $\theta$ and the modulated scalars that denote the sampling magnitude M of global affine transform phase can be obtained as follows:

(1) $$\begin{array}{rl}& \theta ={\mathbb{T}}_{\theta }\left(F_t\oplus F_{t-1}\right)\\ & M={\mathbb{T}}_M\left(F_t\oplus F_{t-1}\right)\end{array}$$where ${\mathbb{T}}_{\theta }$ and ${\mathbb{T}}_M$ denote the transform nodel that generates $\theta$ and M, respectively. These models have the same structure consist of two basic modules with $3\times 3$ convolution kernel, which are trained independently without sharing parameters. $\oplus$ denote the concatenation operation of the features by channel dimension. $\theta =⟨\mathrm{\Delta }{p}_n|n=1,2,3,\dots ,\left|\mathrm{\Re }\right|⟩$ denotes the offset value of each pixel position on the convolution kernel, $\mathrm{\Re }$ denotes a standard 3 $\times$ 3 convolution kernel, which contains nine pixels with the coordinates of $\mathrm{\Re }=⟨\left(-1-1\right)\left(-1,0\right)\left(-1,1\right)\dots \left(0,1\right)\left(1,1\right)⟩$.

Subsequently, we input the spatial offset parameter $\theta$, the modulation scalar M, and the neighboring frame features $F_{t-1}$ to the deformable convolution model $f(\bullet )$ for generating the globally aligned coarse information ${\hat{F}}_{t-1}$, which can be formulated as follows:

(2) $${\hat{F}}_{t-1}=f\left(F_{t-1},\theta ,M\right).$$

For a pixel $p_0$ in the coarse aligned feature vector ${\hat{F}}_{t-1}$ at the rearrangement transformation stage can be formulated as:

(3) $${\hat{F}}_{t-1}\left(p_0\right)=\sum _{p_n\in \mathrm{\Re }}w\left(p_n\right)F_{t-1}\left(p_0+p_n+\mathrm{\Delta }{p}_n\right)M$$where $w\left(p_n\right)$ denotes the weight of pixel $p_n$ in the convolution kernel, and $p_n+\mathrm{\Delta }{p}_n$ indicates the new sampling position of convolution kernel after deformation process modulating.

Local information enhancement block

The global information alignment block achieves coarse calibration to preliminarily rectify spatial shifts or jitter. However, GAB fails to yield satisfactory alignment for some small parts (e.g., hands and feet), which causes this part feature captured from the supporting frame ${\hat{F}}_{t-1}$ to be weakly associated with the current frame $F_t$. Moreover, the model keeps the original geometric structure of the feature during information extraction which is beneficial to improve the robustness of the feature representation and enhance the pose estimation accuracy. So, We further propose a local information enhancement block (LEB) to align local details and produce meticulous fine-tuning at pixel level.

Specifically, taken ${\hat{F}}_{t-1}$ and $F_t$ as inputs, we employ an extended spatial transformation network $\phi$ to computer a transformation matrix $Q\in {\mathbb{R}}^{2\times 2}$ and a translation vector $k\in {\mathbb{R}}^{2\times 1}$ from the position parametric of pixels.

(4) $$Q,k=\phi \left({\hat{F}}_{t-1},F_t\right).$$

During this process, the current frame feature matrix $F_t$ is only used as reference features to participate in the feature propagation process, and their feature values are not changed.

Then, the adaptively trained transformation matrix Q and translation vector $k$ utilized to obtain a set of kernel offset parameters $O=⟨o_1,\dots ,o_9⟩$.

(5) $$O=Q\mathrm{\Re }+k$$where $\mathrm{\Re }$ denotes the $3\times 3$ kernel of conventional convolution.

Subsequently, we achieve the local rearrangement operation of each pixel through the adaptive convolution network to obtain enhanced feature ${\overline{F}}_{t-1}$. Specifically, given the coarse calibrated feature ${\hat{F}}_{t-1}$ and the offset parameters O, the computation of each pixel $q$ in ${\overline{F}}_{t-1}$ can be expressed as follows:

(6) $${\overline{F}}_{t-1}\left(q\right)=\sum _{o_i^q\in O}{w}_i{\hat{F}}_{t-1}\left(q+o_i^q\right).$$For the pixel $q$ in the output features ${\overline{F}}_{t-1}$, the original convolution aggregates the $3\times 3$ pixels (in the kernel location) around $q$ in input features ${\hat{F}}_{t-1}$. In contrast, for each pixel location in the original convolutional kernel, an offset $o$ is added to adjust the sampling (convolution) position in the input features ${\hat{F}}_{t-1}$.

We would like to point out that employing the affine transformation to rearrange information of adjacent frames able to improve the robustness of feature representation and preserve the geometric structure of local details.

Heatmap generation

Finally, we concat all global-local aligned neighboring frame features and the current frame feature at the channel dimension and then employ several residual blocks to aggregate them together to generate improved comprehensive information. We feed them to a detection head to output the heatmaps of each person at the current frame. Note that we utilize a stack of $3\times 3$ convolution layers as the detection head. By effectively capturing closing related information from supporting frames via our dense symmetric temporal alignment learning method, our DSA is more suitable for solving visual degeneration tasks and generating more accurate pose estimation results.

Loss function

The standard pose estimation loss function (Jiao et al., 2022; Liu et al., 2022) is adopted to supervise the learning of final pose estimates. The goal is to reduce the total Euclidean or L2 distance between the estimation and the ground truth heatmaps. The loss function can be formulated as follows:

(7) $$L=\frac{1}{J}\sum _j^J{v}_j\Vert H_j-G_j\Vert$$where $H_j$ and $G_j$ denote the estimated and ground truth heatmap of joint $j$, respectively. $v_j\in (0,1)$ is visibility of joint $j$. The total number of joints in each person is $J=15$.

Experimental settings

Comparison with existing pose estimation methods

PoseTrack2018

We also benchmark our model with other methods on the PoseTrack18 dataset. Quantitive results on the validation set are tabulated in Table 2, including STAF (Raaj et al., 2019), AlphaPose (Fang et al., 2017), TML++ (Hwang et al., 2019), MDPN (Guo et al., 2018), PGPT (Bao et al., 2020), Dynamic-GNN (Yang et al., 2021), and DSA. As shown in this table, our approach once again delivers the best performance. Specifically, we can observe that our model shows a remarkable 78.3 mAP on the validation set, achieving a 0.7 mAP gain over the Dynamic-GNN (Yang et al., 2021). The estimation results of relatively difficult joints, such as elbow, wrist, and ankle, consistently outperformed other methods. These results demonstrate that our model effectively extracts complementary information from adjacent frames and achieves fine-grained feature alignment, significantly enhancing estimation accuracy.

Table 2:

Performance comparisons on the PoseTrack 2018 validation set.

Method	Head	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	mAP
STAF (Raaj et al., 2019)	–	–	–	64.7	–	–	62.0	70.4
AlphaPose (Fang et al., 2017)	63.9	78.7	77.4	71.0	73.7	73.0	69.7	72.5
TML++ (Hwang et al., 2019)	–	–	–	–	–	–	–	74.6
MDPN (Guo et al., 2018)	75.4	81.2	79.0	74.1	72.4	73.0	69.9	75.0
PGPT (Bao et al., 2020)	–	–	–	72.3	–	–	72.2	76.8
Dynamic-GNN (Yang et al., 2021)	80.6	84.5	80.6	74.4	75.0	76.7	71.8	77.6
AlphaPose (Fang et al., 2023)	–	–	–	–	–	–	–	74.7
DSA	81.5	85.1	81.4	74.8	75.6	77.1	72.3	78.3

DOI: 10.7717/peerj-cs.3100/table-2

Furthermore, we display the visual results for complex scenarios of our algorithm in Fig. 2, which demonstrate the importance of our method in embracing complementary cues from neighboring frames. As shown in Fig. 3, our method achieves more robust results in challenging cases such as occlusion and blur.

Badminton dataset

To further evaluate the proposed method, we compare DSA with existing methods in our self-constructed badminton dataset, including CPM (Wei et al., 2016), LSTM-PM (Luo et al., 2018), ResNet-50 (Zhang et al., 2020), HRNet (Sun et al., 2019), and our DSA. The evaluation results are tabulated in Table 3. We observe that HRNet (Sun et al., 2019) has achieved an impressive accuracy of 70.3 mAP, while our proposed method obtains the best performance of 72.8 mAP on this dataset. We also achieve a 79.8 and 78.5 mAP for the Head and the Shoulder joint, respectively. As illustrated in Fig. 4, we display the visual results on the badminton dataset.

Table 3:

Performance comparisons on the badminton dataset.

Method	Head	Shoulder	Elbow	Wrist	Hip	Knee	Ankle	mAP
CPM (Wei et al., 2016)	70.1	61.5	69.0	66.5	55.6	59.8	51.2	61.9
LSTM-PM (Luo et al., 2018)	71.2	63.5	69.2	67.0	57.0	61.2	52.0	63.0
ResNet-50 (Zhang et al., 2020)	72.3	71.0	70.5	69.8	62.1	66.4	57.8	67.1
HRNet (Sun et al., 2019)	76.8	73.2	71.1	70.5	68.6	69.8	59.8	70.3
DSA	79.8	78.5	75.6	71.2	70.8	72.3	61.9	72.8

DOI: 10.7717/peerj-cs.3100/table-3

Ablation study

We perform ablation experiments focused on investigating the contribution of each component in the proposed DSA algorithm, including a global information alignment block (GAB), a local information enhancement block (LEB), and U-Net-like architecture. All experiments are conducted on the PoseTrack2017 validation set. All experiments are performed on the PoseTrack2017 validation set, with results reported in Table 4.