An image quality assessment algorithm based on ‘global + local’ feature fusion

Global-local feature model

This module employs a dual-branch architecture aimed at extracting both global and local features from the distorted and reference images, thereby improving the accuracy of image quality assessment. Specifically, the global feature extraction branch is based on the ViT network, which uses a self-attention mechanism to capture long-range dependencies among image patches across the entire image. This allows the model to capture high-level semantic information and improve the overall understanding of image quality. Specifically, the module reshapes the feature maps output by the ViT network, removing classification tokens and retaining only the information relevant to global image representation. This ensures that the extracted features effectively characterize the overall structure and content of the image. The detailed framework of the global feature extraction process is illustrated in Fig. 2, the framework includes seven main components: location coding, sorting coding, input images, feature pattern extraction, ViT, linear projection, and global feature generation. This module adopts a dual-branch architecture to extract both global and local features from distorted and reference images, aiming to enhance image quality assessment accuracy.

In this process, ViT utilizes a multi-layer Transformer structure to progressively aggregate contextual information among image patches. This enables the model to recognize global patterns within the image and uncover underlying semantic relationships. Such a mechanism not only helps capture long-range dependencies but also enhances the model’s ability to perceive complex distortion patterns, providing a more reliable feature representation for subsequent quality assessment. After processing through ViT, the distorted and reference images obtain their respective global features, denoted as $DFe{a}_{ViT}$ and $RFe{a}_{ViT}$.

Since ViT learns multi-scale feature representations through a multi-layer self-attention mechanism during training, it can effectively process images of varying sizes and resolutions. However, ViT primarily focuses on global semantic information while paying less attention to local image quality details, which may result in insufficient perception of local distortion features. In contrast, CNNs naturally excel at capturing local features and fine details, enabling a more precise analysis of subtle differences in image quality and complementing ViT’s limitations in local feature extraction. To fully leverage the advantages of CNNs in local feature extraction, this module innovatively employs ResNet50 as the local feature extractor and modifies its structure accordingly. Specifically, the final average pooling layer and fully connected layer of ResNet50 are removed to preserve richer spatial information, allowing for better recognition of local structures and texture features in images. This adjustment enables the network to extract more discriminative details, enhancing its understanding of local quality features. After processing through ResNet50, the distorted and reference images obtain local features $DFe{a}_{CNNs}$ and $RFe{a}_{CNNs}$, which retain spatial information. These local features serve as a complement to the global features extracted by the ViT network, providing a more comprehensive representation of image quality and thereby enhancing the accuracy and robustness of the assessment.

Bilinear global-local feature fusion

This module enhances the model’s sensitivity to image quality by preserving both local and global features while integrating fine-grained information. Through this refined fusion, the model is able to focus more on essential visual characteristics such as texture and noise, leading to more precise image quality assessment. Since ResNet50 processes images in a block-wise manner when extracting local features, it inevitably introduces redundant or distortion-irrelevant information, which may interfere with the accuracy of quality evaluation. To address this issue, deformable convolution is employed to adaptively adjust the receptive field of $DFe{a}_{CNNs}$ and $RFe{a}_{CNNs}$. By evaluating the global contribution of each feature and its similarity to the reference image, this mechanism filters out irrelevant and redundant information, thereby capturing the most critical features more precisely, denoted as $DFe{a}_{CNNs}^{\mathrm{\prime }}$ and $RFe{a}_{CNNs}^{\mathrm{\prime }}$. Deformable convolution dynamically samples image features using an offset parameter, which allows the model to reweight feature responses across different regions. The offset values are adaptively learned from the global reference features $RFe{a}_{ViT}$ extracted by ViT. Since $RFe{a}_{ViT}$ encodes the overall structure and key areas of the reference image, it provides crucial guidance on which regions are most significant for image understanding and representation. By adjusting the sampling locations of the convolutional kernels based on these learned offsets, the deformable convolution shifts beyond the constraints of standard grid-based sampling. Instead, it adaptively moves to key regions that align with the global context captured in $RFe{a}_{ViT}$, while simultaneously ignoring irrelevant or redundant areas.

Next, a convolutional network is used to align the refined local features $DFe{a}_{CNNs}^{\mathrm{\prime }}$ with the global distorted features $DFe{a}_{ViT}$, as well as the refined local reference features $RFe{a}_{CNNs}^{\mathrm{\prime }}$ with the global reference features $RFe{a}_{ViT}$. This alignment ensures that both local and global representations are well-integrated, preserving critical image quality information while reducing inconsistencies. After the feature alignment process, the aligned features undergo a feature fusion step, resulting in the final distortion-aware feature representation $DFea$ and $RFea$. These fused features comprehensively encode both global and local information, providing a more holistic representation of image quality. Subsequently, $DFea$ and $RFea$ are processed through a bilinear fusion mechanism to further enhance feature interaction and improve the robustness of the quality assessment. Bilinear fusion effectively models second-order feature relationships, capturing complex dependencies between local and global representations, which helps refine the final quality prediction. The detailed workflow of the bilinear fusion operation is illustrated in Fig. 3; it illustrates the bilinear fusion process, where aligned local and global features undergo bilinear pooling and are mapped into a Euclidean space. This generates fused feature vectors that capture rich interactions for accurate image quality prediction.

The above process not only captures subtle differences between distortion and reference features but also enhances interactions across different feature channels. To begin with, a convolutional operation is applied to the input features $DFea$ and $RFea$, compressing them while preserving essential quality-related information. This compression step effectively reduces computational complexity without sacrificing critical feature representations. Next, a difference operation is performed on the compressed features $DFe{a}^{\mathrm{\prime }}$ and $RFe{a}^{\mathrm{\prime }}$ to obtain the difference feature $DifFea$. This step highlights the discrepancies between the distorted and reference images, allowing the model to detect distortion-related variations more precisely. By emphasizing these differences, the model becomes more sensitive to localized distortions, thereby improving its ability to assess image quality. Following this, bilinear pooling is applied to integrate and refine the extracted features. Specifically, a bilinear dot product operation is conducted between $DifFea$ and $DFe{a}^{\mathrm{\prime }}$, resulting in the fused feature vector $FFea$. This operation facilitates cross-channel interactions, enriching the feature representation and capturing higher-order dependencies between local and global quality cues. The corresponding mathematical formulation is presented in Eq. (1).

(1) $$FFea=AvgPool(DifFe{a}^T\otimes DFe{a}^{\mathrm{\prime }}).$$

$\otimes$ represents the outer product computation, which captures pairwise interactions between feature elements, enabling the model to learn complex relationships between different feature dimensions. This operation enhances the discriminative power of the extracted features by incorporating second-order statistics, which are particularly beneficial for fine-grained quality assessment. Furthermore, $AvgPool()$ denotes the adaptive average pooling operation applied to the feature tensor. This operation dynamically adjusts the spatial resolution of the feature maps, ensuring that the extracted quality representations remain consistent and robust across varying image sizes and resolutions. By aggregating global contextual information while preserving essential quality details, adaptive average pooling helps maintain a balanced representation of local distortions and global structures.

$FFea$ possesses strong representational capabilities, providing a rich and accurate feature representation for subsequent image quality regression. However, since $FFea$ resides in a Riemannian manifold, direct computations on it can be complex and computationally expensive. To simplify subsequent processing and facilitate more efficient calculations, $FFea$ is mapped onto the Euclidean space, denoted as $FFe{a}^{\mathrm{\prime }}$. The transformation follows Eq. (2), where $\odot$ represents element-wise multiplication.

(2) $${FFea}^{\mathrm{\prime }}=\frac{sign(FFea)\odot \sqrt{|FFea|}}{\parallel sign(FFea)\odot \sqrt{|FFea|\parallel }}$$

Gated recurrent unit-based quality aggregation

This module adopts a dual-branch architecture to assess the overall quality of a distorted image in a fine-grained and adaptive manner. One branch predicts the quality score of each image patch, while the other dynamically adjusts the weight of each patch based on variations in image content. The final quality score of the entire image is obtained by aggregating the weighted quality scores of all patches. Specifically, the patch-wise quality prediction branch uses three fully connected layers to independently process the features of each image patch, predict its quality level, and generate the corresponding quality score vector $q$. This design ensures a separate evaluation of each patch’s quality, enabling a more detailed and accurate overall quality assessment. The patch weight computation branch consists of fully connected layers, a gated recurrent unit (GRU) layer, and another fully connected layer. This branch models the temporal relationships and long-term dependencies among image patches, enabling the network to understand the mutual influence and global correlation between patches. During quality assessment, this branch identifies the patches that contribute more significantly to the overall image quality and assigns them higher weights, ensuring a more accurate representation of the perceived image quality. Meanwhile, patches with lower semantic information content are assigned lower weights to reduce their impact on the final quality score. Ultimately, this branch outputs the weight vector $w$ for all image patches. The entire aggregation process is illustrated in Fig. 4, it presents the dual-branch aggregation process, where block-wise quality prediction is combined with adaptive weight computation to generate a final image quality score that reflects both local accuracy and global relevance.

Specifically, the fully connected layer first converts the flattened image quality features $FFe{a}^{\mathrm{\prime }}$ into sequence data suitable for gated recurrent unit processing, representing multiple image patch features as $x$. Then, $x$ is fed into the gated recurrent unit layer to further model the dynamic relationships and global characteristics among image patches. The process of calculation for x is shown in Eqs. (3), (4), (5), (6).

(3) $$updat{e}_i=sigmoid(W_{update}\times [or{i}_{i-1},x_i])$$ (4) $$rese{t}_i=sigmoid(W_{reset}\times [or{i}_{i-1},x_i])$$ (5) $$or{i}_i^{\mathrm{\prime }}=tanh(W_{update}\times [or{i}_{i-1}\times rese{t}_i,x_i])$$ (6) $$or{i}_i=(1\text{-}updat{e}_i)\times or{i}_{i-1}+updat{e}_i\times or{i}_i^{\mathrm{\prime }}.$$

In this model, the gated recurrent unit is utilized to capture the dynamic relationships between image patches, where the update gate and reset gate play a crucial role in controlling the flow and selection of information. Specifically, the update gate determines the extent to which past information is retained while incorporating new information, thereby balancing short-term and long-term dependencies. When the update gate value is close to 1, the network tends to preserve more previous information, whereas a value close to 0 indicates a preference for introducing new input information. The reset gate primarily controls the forgetting mechanism, deciding how much the current state depends on past information. When the reset gate value is close to 0, less past information is retained, and the model focuses more on the current input. Conversely, a higher reset gate value indicates that historical information still important in the current computation. During the computation process, $x_i$ is the $i\text{-}th$ image patch, while $or{i}_i$ is its initial weight, reflecting its relative importance before dynamic adjustment. Additionally, the update and reset gates in the gated recurrent unit structure are linearly transformed by the weight matrices $W_{update}$ and $W_{reset}$, respectively. This enables the model to effectively adjust their contributions within the weight computation branch.

Finally, the gated recurrent unit layer processes the features of all image patches and outputs an initial weight sequence, which reflects the relative importance of each patch in the preliminary computation stage. However, since the raw weight values may have inconsistent scales or information bias, further normalization is required to ensure a reasonable weight distribution suitable for the final quality assessment. Subsequently, this initial weight sequence undergoes a normalization process through the following fully connected layer, transforming it into a standardized attention weight vector $w$. This process not only mitigates the impact of abnormal weights but also ensures that the sum of all patch weights satisfies the required constraints, allowing for a fair comparison of contributions across different image patches. The normalized weight vector $w$ represents the final predicted importance of each image patch, dynamically adjusting their influence in the overall image quality evaluation. The normalization process is provided by Eq. (7).

(7) $$no{r}_{wei}^i=\frac{pr{e}_{wei}^i}{{\sum }_{i=1}^{n}pr{e}_{wei}^i}.$$

In this computation process, $i$ represents the index of each image patch, identifying different regions of the image after block-wise processing. For each image patch, the model first computes its predicted attention weight $pr{e}_{wei}^i$, which reflects its relative importance in the preliminary evaluation stage. However, due to potential variations in scale among different image patches’ predicted weights, directly using these values may introduce bias in the overall quality assessment. To address this, a normalization step is necessary to ensure that all weights are comparable on a global scale and conform to specific numerical constraints. After normalization, the model produces the final normalized attention weight $no{r}_{wei}^i$, which dynamically adjusts each image patch’s contribution to the overall image quality assessment. Compared to the original predicted attention weight, the normalized weight ensures that the total sum of all patch weights adheres to a predefined normalization condition, thereby maintaining a more balanced and reasonable contribution from each image patch in the overall evaluation.

To accurately assess the overall quality score $Quality$ of a distorted image, it is essential to comprehensively integrate the quality information from individual image patches. This ensures that the evaluation result not only aligns with overall visual perception but also captures local quality variations. In this computation process, the model first applies a dot product operation between the image patch quality score vector $qua$ and the image patch weight vector $wei$. This weighted summation effectively merges the quality scores of all patches into a single overall score. Specifically, $qua$ represents the independent quality assessment results of each image patch, while $wei$, as the normalized attention weight, determines the influence of each patch on the final quality score. The computation is given in Eq. (8), where $qu{a}_i$ is the quality prediction score of the $i\text{-}th$ image patch.

(8) $$Quality=\sum _{i=1}^{n}we{i}_i\times qu{a}_i=\frac{{\sum }_{i=1}^{n}pr{e}_{wei}^i\times qu{a}_i}{{\sum }_{i=1}^{n}pr{e}_{wei}^i}$$

Dataset and parameters

To comprehensively evaluate the effectiveness of the IQA-GL, this article used public image quality assessment datasets, namely CSIQ, TID2013 and KADID-10k, as illustrated in Figs. 5, 6 and 7. These datasets encompass a variety of typical distortion types and include extensive subjective quality scores, providing a reliable benchmark for evaluating the effectiveness of the proposed approach. CSIQ consists of 30 natural reference images distorted using six different types of distortions at varying intensities, yielding 866 distorted images in total. Subjective quality scores were collected using a pairwise comparison protocol, normalized between 0 and 1. TID2013 provides 25 reference images, each degraded with 24 different distortion types at five intensity levels, resulting in a total of 3,000 distorted images. It offers mean opinion scores collected from multiple human subjects, making it one of the most comprehensive IQA datasets available. KADID-10k contains 81 high-quality reference images, with distortions applied across 25 types and five levels each, leading to 10,125 distorted images. The subjective ratings were gathered via a large-scale crowdsourcing platform, ensuring statistical reliability for data-driven IQA model training. By evaluating the proposed method on these three datasets, its applicability and robustness under various distortion types can be comprehensively verified.

To assess the generalization ability of the proposed IQA-GL model, we performed a cross-database evaluation. Specifically, the model was trained on the CSIQ dataset and tested on TID2013 and KADID-10k, and vice versa. The results showed that although there was a slight performance drop compared to intra-dataset testing, IQA-GL maintained competitive accuracy across different datasets. This demonstrates the model’s robustness and adaptability to various distortion types and data distributions.

In this experiment, to enhance the model’s generalization ability, each training image underwent normalization and random horizontal flipping, followed by a random crop to a fixed size of 224 $\times$ 224 pixels before being fed into the network for training. The validation set was used to fine-tune network parameters and optimize performance, while the test set was employed to evaluate the model’s capability in assessing image quality. For the dual-branch feature extraction module in the proposed IQA-GL method, the convolutional neural network (CNN)-based feature extraction branch utilized ResNet-50, while the transformer-based feature extraction branch adopted ViT-B-8 as the backbone network. The model was trained using the mean squared error loss function, with AdamW as the optimizer. To ensure efficient learning, a cosine annealing scheduler was applied to dynamically adjust the learning rate for each parameter group. The cosine annealing learning rate scheduler was chosen due to its effectiveness in gradually reducing the learning rate in a smooth, non-linear fashion. This approach helps the model escape local minima during early training while allowing fine-tuning in later stages. The total number of training iterations was set to 250 to ensure stable convergence and optimal performance. The decision to set the number of training iterations to 250 was based on empirical observation of convergence behavior across multiple experimental runs. We found that the model typically converged within this range, and extending the iterations further led to marginal improvements while increasing the risk of overfitting. To support this choice, a training/validation loss curve showing stable convergence within 250 iterations under the cosine annealing schedule is shown in Fig. 8. Some hyperparameters used in the experiments is shown in Table 1. In comparison, traditional methods such as BRISQUE is lightweight with near real-time performance but show significantly lower accuracy on AI-generated image quality tasks. Deep learning-based methods like MUSIQ tends to fall between our models in terms of computational cost but still underperform in the AIGI scenarios we evaluate. BRISQUE is a fast, traditional algorithm with very low computational cost but limited prediction accuracy. MUSIQ adopts a patch-based transformer architecture that achieves moderate performance with relatively low resource consumption. ResNet-50 offers a good balance between accuracy and efficiency, making it suitable for most practical scenarios. ViT-B-8, while delivering the best accuracy among all models, requires significantly higher computational resources. These comparisons suggest that while our method incurs higher computational costs, it offers superior performance and can be optimized in the future via compression techniques for deployment in resource-constrained settings. The comparison of computational complexity is shown in Table 2.

This experiment selected ResNet-50 and ViT-B-8 as the backbone architectures based on their complementary characteristics and proven effectiveness in prior related studies. ResNet-50 is a well-established convolutional neural network with strong representational power and robust generalization ability, particularly on natural scene images. It serves as a representative of traditional CNN-based models, which are efficient in learning local texture features. ViT-B-8, a variant of the Vision Transformer, uses smaller patch sizes and is capable of capturing long-range dependencies and global structures, which are especially relevant for AI-generated images that often exhibit unique global patterns. ViT-B-8 is selected due to its relatively balanced computational cost and superior performance in transformer-based vision tasks.

Although more and more studies use efficientNet and Swin Transformer, such as Koonce (2021) and Liu et al. (2021). In this experiment, efficientNet achieves high accuracy on natural images, it performs less stably when transferred to AI-generated image assessment tasks. Swin Transformer, although powerful, introduces higher complexity and latency without a significant improvement in our specific evaluation setting.

Contrast experiment

A comparison was made among six advanced image quality assessment methods, including the proposed IQA-GL. The experimental results are shown in Table 3.

PLCC quantifies the linear relationship between the predicted scores and the subjective quality scores (MOS or DMOS), with values ranging from $-1$ to $1$. A PLCC value closer to $1$ indicates a stronger positive correlation, while values near $-1$ or $0$ imply a negative or no correlation, respectively. The PLCC is calculated as Eq. (9):

(9) $$\mathrm{P}\mathrm{L}\mathrm{C}\mathrm{C}=\frac{{\sum }_{i=1}^N(x_i-\overline{x})(y_i-\overline{y})}{\sqrt{{\sum }_{i=1}^N{(x_i-\overline{x})}^2}\sqrt{{\sum }_{i=1}^N{(y_i-\overline{y})}^2}}.$$

On the CSIQ dataset, the PLCC of the IQA-GL is 0.21% higher than that of the second-best method. On the TID2013 dataset, the proposed method achieves a 0.31% improvement in PLCC, indicating a stronger linear correlation between the predicted scores and the subjective evaluations. On the KADID-10k dataset, the proposed method also achieved the best results.

SRCC measures the monotonic relationship between predicted quality scores and ground truth subjective scores. Unlike PLCC, SRCC does not require a strictly linear correlation but instead assesses whether the ranking trend of predicted scores is consistent with human perception. The SRCC is calculated as Eq. (10):

(10) $$\mathrm{S}\mathrm{R}\mathrm{C}\mathrm{C}=1-\frac{6{\sum }_{i=1}^N{d}_i^2}{N(N^2-1)}$$where:

• $d_i=\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}(x_i)-\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}(y_i)$ represents the difference in ranking between the predicted score and the MOS,

• $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}(x_i)$ and $\mathrm{r}\mathrm{a}\mathrm{n}\mathrm{k}(y_i)$ are the ranks of the predicted score and MOS, respectively,

• N is the total number of samples.

On the CSIQ dataset, the proposed method achieves a 0.1% improvement in SRCC compared to the second-best method, while on the TID2013 and KADID-10k datasets, the SRCC increases by 0.1%. These results suggest that the proposed method more accurately captures the perceptual ranking of image quality.

The experimental results exhibit strong adaptability and robustness in handling various distortion types and scenarios, proving its superiority in image quality assessment tasks.

Ablation experiment