A quality assessment algorithm for no-reference images based on transfer learning

Multi-scale semantic information feature extraction module

In the field of deep neural networks, in order to obtain multi-scale semantic features, there have been many mature studies based on the strategy of extracting information from different convolutional layers of images.

AlexNet-NDTL (Kollem et al., 2023) is based on an improved AlexNet and uses network-based deep transfer learning to extract features from the dataset. Updated GoogLeNet (Yang et al., 2023) replaces the $7\times 7$ convolution kernel of the first layer with three $3\times 3$ convolution kernels based on GoogLeNet, adds the ECA attention mechanism to the inception module, and uses a residual network to connect the E-inception module, solving the problem of increased information loss and gradient loss due to increased network depth. Improved ResNet-50 (Wu et al., 2023a) introduces the Squeeze-and-Excitation attention mechanism to improve the residual unit of ResNet-50, and combines the Swish function and Ranger optimizer to ensure the effectiveness of feature learning and training, further improving model performance. Improved ResNet-50 has a balanced performance compared with typical recognition algorithms such as AlexNet-NDTL (Kollem et al., 2023), Updated GoogLeNet (Yang et al., 2023), VGG-16 (Sharma & Guleria, 2023), ResNet-18 (Sunnetci et al., 2023), and DenseNet-201 (Salim et al., 2023). It explicitly constructs an identity mapping. The residual network introduces skip connections, which reduces the possibility of gradient vanishing and gradient exploding during training, and improves the performance of deep networks. The flowchart of feature extraction for multi-scale semantic information is shown in Fig. 2.

The input distorted image information $x$ first passes through the first convolutional layer weight layer to obtain the preliminary mapping function $F(x)$, and then reaches the weight layer based on the second convolutional layer through the linear rectification function. The final output of the weight layer is added to the shallow information of the jump connection to obtain the final mapping function $F(x)+x$. This structure can directly map shallow information to deep positions, thus improving the accuracy of the model.

This article innovatively extracts the convolutional layer information $inf{o}_1$, $inf{o}_2$ and $inf{o}_3$ in the backbone network convolutional layer $c\mathrm{\_}laye{r}_{2\to 10}$ (representing the tenth operation of the second convolutional layer), $c\mathrm{\_}laye{r}_{3\to 12}$ and $c\mathrm{\_}laye{r}_{4\to 18}$ as local semantic features of the image, and the convolutional layer information $inf{o}_4$ in $c\mathrm{\_}laye{r}_{5\to 10}$ as the global semantic feature of the image. The method of obtaining multi-scale feature $inf{o}_i$ is shown in Eq. (1).

(1) $$inf{o}_i=Res(x,{\delta }_1)(i=1,\dots ,4).$$

Among them, $inf{o}_i$ represents the input image obtaining features of four different scales through the ResNet network, $Res()$ represents the feature extraction process of the ResNet-50 network, $x$ represents the information of the input image, and ${\delta }_1$ represents the parameters of the ResNet-50 network. The fusion of multi-scale semantic features enables the complementary fusion of deep feature information and shallow feature information, which improves the accuracy of the model to a certain extent.

Visual perception module

Because the traditional NR-IQA method based on deep learning network fails to fully incorporate the characteristics of the human visual perception system when designing the network structure, it leads to problems of insufficient sample number and limited sample diversity. This method often relies on limited data sets for training, which not only affects the generalization ability of the model, but also causes a large gap between the assessment results and human perception results. Therefore, to address these problems, this article innovatively introduces a reference-free image quality assessment method based on perceptual semantic features.

In this new approach, this article uses semantic features to describe the quality of images, and closely integrates the semantic content of images with quality assessment during the quality assessment process. In this way, not only can the human perception process of images be better simulated, but the automatic assessment process can also be made closer to human understanding and assessment of the semantic content of images. This improvement is expected to significantly improve the generalization ability of the model, making its performance on different samples more stable and reliable, and further narrowing the gap between automatic assessment results and human subjective perception. This article introduces a perception module to capture the information of distorted images. The calculation method of the perception module is shown in Eq. (2).

(2) $$inf{o}_i^{\mathrm{\prime }}=per(inf{o}_i,{\delta }_2)(i=1,\dots ,4).$$

Among them, $inf{o}_i^{\mathrm{\prime }}$ represents the features of four different scales after being processed by the perception network model, $per()$ represents the perception network model, $inf{o}_i$ represents the features of four different scales obtained by the input image through the ResNet network, and ${\delta }_2$ represents the network parameters of the perception module. The network structure of this perception module is shown in Fig. 3.

Firstly, the local semantic features $inf{o}_i(i=1,2,3)$ and the global semantic features $inf{o}_4$ output by the ResNet-50 network are divided into non-overlapping modules. Secondly, these non-overlapping modules are connected and stacked according to the number of dimensions of their respective channels. Finally, a 1 * 1 convolution operation and a global average pooling operation are performed to obtain the feature $inf{o}_i^{\mathrm{\prime }}(i=1,\dots ,4)$ processed by the perception module. Global average pooling calculates the mean of the two-dimensional feature map of each channel and takes the average of all pixel values of each channel as the output value of the channel. This operation greatly simplifies the representation of the data, thereby effectively reducing the number of parameters of the model. In this way, the complexity of the model is reduced and the risk of overfitting is effectively suppressed. In addition, this method can also preserve the spatial information of the input image, making the model more flexible and adaptable when processing inputs of different sizes.

Adaptive fusion network module

In the feature fusion stage, a large amount of redundant information is usually inevitably introduced, which will interfere with the accurate assessment of image quality. In order to solve this problem and reduce the impact of noise superposition in different feature maps, this article applies an adaptive feature fusion module to the feature $inf{o}_i^{\mathrm{\prime }}(i=1,2,3)$ processed by the perception module. The adaptive feature fusion module assigns higher weights to the key parts of each layer of features when performing feature fusion. Specifically, the adaptive fusion module can dynamically adjust the fusion ratio between different layers according to the importance of feature maps. This adaptive adjustment mechanism can not only highlight important feature information, but also effectively suppress unnecessary features, thereby optimizing the entire feature fusion process.

Through this approach, this article not only enhances the emphasis on important features, but also significantly reduces the amount of parameters in the network model. This reduction in the number of parameters further improves the efficiency of the model and makes the assessment of image quality more accurate and reliable. The network structure of this adaptive fusion network module is shown in Fig. 4.

The input of the network is the feature $inf{o}_i^{\mathrm{\prime }}(i=1,2,3)$ processed by the perception module. These features have different dimensions, where $inf{o}_1^{\mathrm{\prime }}$ contains a 16-dimensional feature map, $inf{o}_2^{\mathrm{\prime }}$ contains a 32-dimensional feature map, and $inf{o}_3^{\mathrm{\prime }}$ contains a 64-dimensional feature map. Since the dimensions of these feature maps are inconsistent, in order to perform effective feature fusion in the model, this article converts them into feature maps of the same size. This model uniformly converts all feature maps into 64-dimensional feature maps. Specifically, this article uses 64 convolution kernels of size $1\times 1$ to convolve these three features. The step size of each convolution kernel is different to ensure that different information in each feature map can be captured during the feature fusion process. In this way, this article can better fuse feature maps of different dimensions, so that the model can make full use of these features and improve the overall performance and accuracy. This feature fusion method not only ensures the consistency of feature map dimensions, but also can capture more local and global information through convolution operations of different step lengths, thereby enhancing the feature expression ability of the model. The features after dimension matching are summed up, as shown in Eq. (3).

(3) $$inf{o}_{sum}^{\mathrm{\prime }}=M(inf{o}_1^{\mathrm{\prime }})+M(inf{o}_2^{\mathrm{\prime }})+M(inf{o}_3^{\mathrm{\prime }}).$$

Among them, $inf{o}_{sum}^{\mathrm{\prime }}$ represents the features obtained after fusion, $M()$ represents the dimension matching function, and $inf{o}_i^{\mathrm{\prime }}(i=1,2,3)$ represents the features processed by the perception module. The size of the fused feature map is $height\ast width\ast channel$ (where $height$ represents the height of the feature map, $width$ represents the width of the feature map, and $channel$ represents the number of channels of the feature map). Then, global average pooling is used to capture the global information of $inf{o}_i$. $inf{o}_i$ represents the features of four different scales obtained by the input image through the ResNet network, and the features are aggregated into a $channel\ast 1$-dimensional feature vector $inf{o}_{eig}$. Then, based on the fully connected layer, $inf{o}_{eig}$ is compressed into $(channel/coe)\ast 1$-dimensional squeezed feature information $inf{o}_{ext}$, where $coe$ is the compression coefficient, as shown in Eq. (4).

(4) $$inf{o}_{ext}=A(matri{x}_{(channel/coe)\ast 1}inf{o}_{eig}).$$

Among them, $inf{o}_{ext}$ represents the squeeze feature information, $A()$ represents the activation function, $matri{x}_{(channel/coe)\ast 1}$ represents the $(channel/coe)\ast 1$ dimensional weighted matrix, and $inf{o}_{eig}$ represents the $channel\ast 1$ dimensional feature vector.

The model complexity can be reduced to a certain extent by the squeezing operation of the fully connected layer. Specifically, feature $inf{o}_{ext}$ is input to the $softmax$ layer and the fully connected layer, which can adaptively select important features from different scales. In this process, $sof{t}_i(i=1,2,3)$ represents the weight vector obtained through the adaptive network, which corresponds to feature selection at different scales. To be more specific, assume that $sof{t}_i(i=1,2,3)$ is the weight vector generated by the adaptive network at different scales, and $sof{t}_i^j(i=1,2,3)$ represents the $j-th$ element in $sof{t}_i(i=1,2,3)$, respectively. Due to the inherent properties of the $softmax$ layer, the elements of the weight vector $sof{t}_i(i=1,2,3)$ will satisfy the conditions of ${\sum }_{i=1}^{3}sof{t}_i^j=1$. This means that at each position $j$, the sum of the elements of the weight vector $sof{t}_i(i=1,2,3)$ is 1, ensuring the normalization of the weights and the reasonable distribution of relative importance. Finally, by operating the information with weights at different scales, the final feature map $f_{fused}$ can be obtained, as shown in Eq. (5).

(5) $$f_{fused}=\sum _{i=1}^{3}sof{t}_{i}inf{o}_i^{\mathrm{\prime }}.$$

Among them, $f_{fused}$ represents the final feature map, $sof{t}_i(i=1,2,3)$ represents the weight vector generated by the adaptive network at different scales, and $inf{o}_i^{\mathrm{\prime }}(i=1,2,3)$ represents the features processed by the perception module. This method can not only effectively reduce the complexity of the model, but also improve the feature extraction ability and generalization performance of the model by adaptively selecting and combining important features at different scales.

Quality assessment regression module

The target network of image quality assessment aims to accurately map the learned multi-scale image features to the corresponding quality scores. In this model structure, the target network used is composed of two fully connected layers. These two fully connected layers are responsible for fusing and processing information at different levels to ensure that the final output quality score can effectively reflect the quality of the input image. Specifically, the target network first receives the global semantic information $inf{o}_4$, which contains a high-level understanding of the overall characteristics of the input image. At the same time, the target network also receives the feature $f_{fused}$ obtained through the adaptive fusion network. The role of the adaptive fusion network is to intelligently combine multi-scale image features to generate more representative feature representations. Then, through Eq. (6), the target network connects the global semantic information $inf{o}_4$ with the feature $f_{fused}$ to obtain the fused feature representation $f_{fused}^{\mathrm{\prime }}$. This process not only retains the global information of the image, but also integrates multi-scale features, so that the final $f_{fused}^{\mathrm{\prime }}$ can more comprehensively reflect the quality characteristics of the image. Finally, the target network maps $f_{fused}^{\mathrm{\prime }}$ to a specific quality score through further calculation, thus completing the image quality assessment task.

(6) $$f_{fused}^{\mathrm{\prime }}=F(f_{fused},inf{o}_4^{\mathrm{\prime }}).$$

Among them, $f_{fused}^{\mathrm{\prime }}$ represents the fused features, $F()$ represents the feature fusion function, $f_{fused}$ represents the features obtained by the adaptive fusion network, and $inf{o}_4^{\mathrm{\prime }}$ represents the features processed by the perception module.

After the obtained feature $f_{fused}^{\mathrm{\prime }}$ is used as the input of the target network, it is further processed. This article selects the $LeakyReLU$ function as the activation function of the target network. The $LeakyReLU$ function is a commonly used activation function that allows a certain negative slope when the input value is negative, thereby alleviating the “neuron death” problem that may occur in the $ReLU$ function to a certain extent. This target network inputs the feature $f_{fused}^{\mathrm{\prime }}$ into fully connected layers activated by the $LeakyReLU$ activation function, which perform nonlinear transformations on the input features to capture important patterns and features required in image quality assessment. Finally, after multiple layers of processing, the target network outputs an image quality score, which represents a comprehensive assessment result of the input image quality. This quality score is shown in Eq. (7), which specifically describes the calculation process from feature $f_{fused}^{\mathrm{\prime }}$ to the final quality score. Through this process, the network can effectively use the learned features to evaluate the image quality and return a quantitative quality score S.

(7) $$S=Eva(f_{fused}^{\mathrm{\prime }},weigh{t}_{full},bias).$$

Among them, S represents the final predicted image quality score, $Eva()$ represents the quality assessment network model, $f_{fused}^{\mathrm{\prime }}$ represents the fused features, $weigh{t}_{full}$ represents the fully connected layer weight of the target network, and CC represents the fully connected layer $bias$ of the target network.

Based on the above content, the overall network model proposed in this article is shown as Eq. (8).

(8) $$S_{final}=T\mathrm{\_}Eva(x,inf{o}_i,inf{o}_i^{\mathrm{\prime }},S)(i=1,2,3)$$

Among them, $S_{final}$ represents the final quality score of the image, $T\mathrm{\_}Eva()$ represents the entire network model, $x$ represents the input distorted image, $inf{o}_i$ represents the multi-scale feature extraction network model from ResNet-50, $inf{o}_i^{\mathrm{\prime }}$ represents the perception network model, S represents the quality assessment network model, and $i$ represents the features of $i$ different scales after processing by the perception network model.

Dataset introduction and data augmentation

Due to the limited availability of datasets in the field of image quality assessment, acquiring sufficient experimental data can be challenging.

1. LIVE

This dataset contains numerous images and videos with a variety of distortions, including compression artifacts, noise, blur, and transmission errors. Like the KonIQ-10K dataset, each image and video in the LIVE dataset is rated subjectively by human observers.For the LIVE dataset, the process includes:

• Random Sampling: Selecting various image samples to increase data variability.

• Horizontal Flipping: Enhancing data diversity through mirroring.

• Resizing: Adjusting images to a uniform size of 224 × 224 pixels.

• Format Conversion and Normalization: Ensuring consistency in data format and distribution.

2. TID2013

The TID2013 dataset is one of the most comprehensive datasets used for image quality assessment, containing images with 24 types of distortions at five different levels of severity. For the TID2013 dataset, the process includes:

• Resizing: All images are resized to $384\times 384$ pixels to ensure uniformity across the dataset.

• Data Augmentation: Techniques such as random cropping and flipping are applied to increase data variability and model performance.

• Format Conversion and Normalization: Consistent with other datasets, this step ensures that all images are in the same format and normalized to a standard distribution.

3. KonIQ-10K

This dataset is widely utilized for image quality evaluation. It includes over 10,000 images gathered from various online platforms, covering a range of shooting conditions and devices. For the KonIQ-10K dataset, the process includes:

• Resizing: Adjusting images to 512 × 384 pixels.

• Random Sampling and Horizontal Flipping: Similar to the LIVE dataset, to enrich the dataset.

• Data Format Conversion and Normalization: Consistent with the preprocessing steps used for the LIVE dataset.

4. BIQ2021

The BIQ2021 dataset is a more recent addition to the field of image quality assessment, specifically focusing on blind or no-reference image quality tasks. It includes a diverse range of images, featuring both natural and distorted content. For the BIQ2021 dataset, the process includes:

• Resizing: Images are resized to $256\times 256$ pixels to maintain consistency.

• Random cropping and horizontal flipping: These techniques are used to augment the dataset and improve model robustness.

• Format conversion and normalization: Ensures consistency in data format and pixel value distribution across all samples.

The code used in this experiment has been made publicly available on GitHub. You can access it via the following link: https://github.com/deyi2024/A-Quality-Assessment-Algorithm-for-No-Reference-Images-Based-on-Transfer-Learning.git.

Dataset with manual scoring

Unlike NSIs, which are captured from real scenes, AGIs are produced by AI models and exhibit unique quality characteristics. Consequently, viewers assess AGIs differently from NSIs. To address this, Zhang et al. (2023c) introduced the AGIQA-1K database, which features 1,080 AGIs with quality labels covering technical issues, AI artifacts, unnaturalness, discrepancy, and aesthetics.

Comparison based on the real distortion dataset $KonIQ-10K$ and $BIQ2021$

The research method in this article was compared with the assessment methods proposed by other scholars on the same datasets ( $KonIQ-10K$ and $BIQ2021$). First, 90% of all target images in the dataset are randomly selected as training data, and the remaining 10% are used as test data. In order to ensure the robustness of the experimental results and reduce the accidental effects caused by data division, the division of the training set and the test set is repeated 50 times, and the median of all results is taken as the final experimental result. In order to reduce the impact of outliers on model training, the mean absolute error (MAE) was used as the loss function during the training process. MAE can minimize the gap between the model prediction results and the actual subjective quality score, making the model’s prediction results as close to the subjective quality score as possible. The calculation method of the loss function is shown in Eq. (11). By minimizing this loss function, the model can more accurately reflect the true quality of the image.

(11) $$Loss=\frac{{\sum }_{i=1}^{Sum}||T\mathrm{\_}Eva(piec{e}_i,inf{o}_i,inf{o}_i^{\mathrm{\prime }},S)-Q\mathrm{\_}rea{l}_i||}{Sum}.$$

Among them, $Loss$ represents the loss function value, $Sum$ represents the total number of image blocks, $T\mathrm{\_}Eva()$ represents the entire network model, $piec{e}_i$ represents the $i-th$ image block of the input distorted image, $inf{o}_i$ represents the multi-scale feature extraction network model from $ResNet-50$, $inf{o}_i^{\mathrm{\prime }}$ represents the perception network model, S represents the quality assessment network model, and $Q\mathrm{\_}rea{l}_i$ represents the true quality score of the $i-th$ image block.

This experiment compares the image quality assessment methods based on manually extracted image features, such as the IQA—NRTL method proposed in this article, the ECGQA algorithm proposed by Liu et al. (2021), the DLBF algorithm proposed by Mahum & Aladhadh (2022), the FDD algorithm proposed by Chen, Rottensteiner & Heipke (2021), and the image quality assessment methods based on deep learning, such as the CELL algorithm proposed by Rasheed, Shi & Khan (2023), the PDAA algorithm proposed by Valicharla et al. (2023), and the CFFA algorithm proposed by König et al. (2024). The experiments are carried out on the authentically distorted images from $KonIQ-10K$ and $BIQ2021$ dataset, and the algorithm effects are compared with the PLCC and SROCC values, as shown in Table 1.

As can be seen from Table 1, the assessment methods based on manual feature extraction, such as ECGQA, DLBF, and FDD, perform poorly on real distortion datasets, which is specifically reflected in the low PLCC and SROCC values and the inability to accurately predict the image quality score. This shows that these methods have obvious limitations when dealing with complex image quality assessment tasks and cannot effectively capture the true quality characteristics of images. In contrast, deep learning-based assessment methods such as CELL, PDAA, and CFFA perform much better on real distortion datasets. These methods learn image features through deep neural networks and improve the accuracy of image quality prediction to a certain extent. However, these methods still have difficulty in achieving ideal prediction accuracy for various reasons. The CELL method lacks a module that captures local semantic information, which makes it unable to fully understand the local quality changes of the image; although the PDAA method considers the importance of local information for global quality prediction, it ignores the correlation between the extracted hierarchical features, which limits its performance improvement; the adaptive layer of the CFFA method ignores important underlying distortion information when selecting features, making its prediction results imperfect.

The IQA—NRTL method proposed in this article improves the above shortcomings. IQA—NRTL is designed to pay more attention to the comprehensiveness and accuracy of feature extraction, especially in capturing local semantic information and processing multi-layer feature correlation. Experimental results show that the PLCC and SROCC values of the IQA—NRTL method on the real distortion data set have reached the optimal level, significantly better than traditional manual feature extraction methods and existing deep learning methods. Among them, in terms of PLCC, the IQA—NRTL method improves by 15.64% and 12.68% respectively compared with the optimal solutions of the first two strategies. In terms of SROCC, the IQA—NRTL method improves by 27.76% and 5.65% respectively compared with the optimal solutions of the first two strategies. This shows that the IQA—NRTL method has high accuracy and reliability in evaluating data sets of real natural scenes.

Comparison based on artificial distortion dataset LIVE and $TID2013$

In order to verify the performance of this method on a synthetic dataset, this article selected the LIVE and $TID2013$ dataset, which is widely used in the field of image quality assessment. Through this dataset, this article systematically compared and analyzed the proposed assessment method with other commonly used quality assessment methods. Specifically, the PLCC and SROCC values of each method on the LIVE dataset were calculated to measure the accuracy and consistency of different methods in image quality prediction. Table 2 shows the PLCC and SROCC values of each method on the LIVE data set. It can be clearly seen from the results that the IQA—NRTL method proposed in this article performs well on both key performance indicators, significantly better than traditional manual feature extraction methods and existing deep learning methods.

From the data in Table 2, it can be concluded that the assessment method IQA—NRTL proposed in this article has achieved relatively ideal test results on the LIVE dataset, and its PLCC and SROCC values are better than most methods. Although the assessment score of the IQA—NRTL method did not reach the best among all mainstream image quality assessment methods, its PLCC index reached 0.914, which was only 0.006 lower than the best algorithm, and its SROCC index reached 0.929, which was only 0.009 lower than the best method. This shows that the IQA—NRTL method can provide reliable quality prediction results in practical applications while maintaining high accuracy and consistency. Compared with the mainstream FDD and PDAA methods, the performance of the IQA—NRTL method in practical applications is not substantially different, and it can meet the needs of actual image quality assessment and achieve ideal quality prediction results. In summary, by comparing the experimental results, it can be seen that the IQA—NRTL method proposed in this article performs quite well on the LIVE dataset. Although it is slightly inferior to the best method in some indicators, its overall performance is comparable to the current mainstream image quality assessment methods.

Fusion experiment

To further verify the effectiveness of the IQA—NRTL method proposed in this article, this article conducted multiple comparative experiments on the $KonIQ-10K$, $BIQ2021$, LIVE and $TID2013$ dataset. First, this article uses a model containing only the $ResNet-50$ network as the baseline method for experiments, named $Exp\mathrm{\_}ResNet-50$; then, the multi-scale semantic information feature extraction module, visual perception module and adaptive fusion network module are added in turn for experiments, named $Exp\mathrm{\_}Semantic$, $Exp\mathrm{\_}Vision$ and $Exp\mathrm{\_}Adapt$ respectively; finally, the IQA—NRTL model containing all modules is used for experiments. Through these experiments, the experimental results of the five models were obtained and their PLCC and SROCC values were compared. The experimental results are shown in Table 3.

Through the above experimental steps, we can compare in detail the effect of each module on improving model performance, and finally verify the effectiveness and superiority of the IQA—NRTL method proposed in this article in image quality assessment. Experimental results show that the addition of each module significantly improves the performance of the model, and the final IQA—NRTL model performs best in both PLCC and SROCC indicators.

Subjective experiment based on $AGIQA-1K$

Both the algorithms based on “Hand-Crafted Statistics” and “Deep Learning,” as well as the IQA-NRTL algorithm proposed in this article, achieved satisfactory results in image quality assessment experiments, as shown in Table 4. However, in subjective evaluation experiments, the results of the IQA-NRTL algorithm were closer to human visual scores compared to other algorithms. This indicates that the proposed algorithm demonstrates a clear advantage in subjective evaluations, further underscoring its ability to better simulate the human visual system’s perception of image quality. This advantage highlights the superior practical reliability of the IQA-NRTL algorithm in image quality assessment tasks.