A full reference quality assessment method with fused monocular and binocular features for stereo images

FR-SIQA

Most early FR-SIQA methods directly applied the 2D-IQA method to independently process each view image to predict the stereoscopic image quality without considering the binocular vision characteristics. However, Chen et al. (2013) pointed out that using the 2D-IQA method to predict the quality of stereoscopic images has better results in the case of symmetrical distortion, but poorer results in the case of asymmetrical distortion. Symmetric distortion means that the left and right views have the same degree of distortion, while asymmetric distortion means that the left and right views have different types of distortion or different degrees of distortion. Therefore, people began to consider the characteristics of binocular vision to solve the problem of asymmetric distortion. Shao et al. (2013) proposed a new FR-SIQA method which first classifies stereo images into non-corresponding areas, binocular fusion and suppression areas, then extracts quality-sensitive features from reference and distorted stereo images; finally through Each area is evaluated independently taking into account its binocular perceptual characteristics, and all evaluation results are integrated into an overall score. Fezza & Larabi (2014) proposed to use local entropy as a weight factor to combine the left and right views to synthesize the middle view, and the performance of this method was significantly improved. Geng et al. (2016) used the independent component analysis method to extract the features of stereoscopic images, and combined their feature similarity and local brightness consistency to construct a FR-SIQA. Ma et al. (2017) introduced a binocular perception model based on SSIM, which jointly considers the quality of cyclopean images and differential images to predict the perceptual quality of stereoscopic images. Chen & Zhao (2019) proposed a full reference method that combines local and global visual features to perceive the quality of stereoscopic images. Si et al. (2021) first proposed a SIQA method to divide stereo images into occluded and non-occluded areas. First, the disparity information and the Euclidean distance between stereo pairs are used to find a segmentation strategy for occluded and non-occluded areas in the scene; the occluded areas represent monocular vision, and the unoccluded areas show binocular vision of HVS. Global and local features are then extracted from the regions and used to predict visual quality. Ahmed Seghir et al. (2023) calculated the binocular summation map by adding the left and right images of the stereo pair, and then used an improved distortion pixel measure based on gradient similarity to evaluate the quality of the binocular summation map.

RR-SIQA

Yang et al. (2018) proposed a RR-SIQA method based on sparse coding. Wan, Gu & Zhao (2019) proposed a RR-SIQA method that uses sparse representation and natural scene statistics to simulate the brain's visual perception. The distribution statistics of the extracted categorical visual primitives are sparsely represented to measure the visual information, and the natural scene statistics of the local normalized brightness coefficients are used to evaluate the natural loss due to the presence of distortion. The quality score is calculated by calculating the difference between visual information and natural scene statistics for the original and distorted images, and then using a prediction function trained by support vector regression. Ma, Xu & Han (2021) proposed a RR-SIQA method based on gradient-based sparse representation and structural degradation. The proposed method is based on two main tasks: the first task extracts distribution statistics of visual primitives through gradient sparse representation, while the second task measures the distribution of visual primitives by extracting joint statistics of gradient magnitude and Laplacian of Gaussian features due to Structural degradation of stereoscopic images caused by distortion.

NR-SIQA

In current research, NR-SIQA methods can generally be categorized into those based on traditional machine learning and those based on deep learning.

The first category typically relies on natural scene statistics (NSS) features, which are then input into machine learning methods such as support vector regression (SVR) to obtain quality assessment scores. Messai, Hachouf & Seghir (2020) introduced a novel NR-SIQA method. They first considered the presence of binocular rivalry, generating an intermediate view from the left and right views, and then extracted features like gradient magnitude, relative gradient direction, and relative gradient magnitude from the intermediate view. They employed AdaBoost regression to achieve the final quality assessment score. Zhang et al. (2022) proposed a multi-scale perceptual feature-based SIQA method. Stereoscopic images were transformed using a pyramid to obtain sensitized images of varying resolutions. These images were further processed to extract structural features, intrinsic structure features, and depth structure features. A genetic algorithm-based SVR (GA-SVR) was employed to achieve the final quality score. However, due to limitations such as high complexity and imprecise regression models associated with manually extracted features, these traditional NR-SIQA methods cannot achieve optimal consistency with HVS. Shen et al. (2023) proposed a natural scene statistical-based NR-SIQA method. Firstly, two-dimensional features are extracted from monocular images in spatial and transform domains. Then, perception-related features associated with three-dimensional quality are extracted from stereo visual perception information. Following this, through natural scene statistical modeling and adaptive principal component analysis feature pruning, the optimal regression model is fitted using SVR.

With the advancement of deep learning techniques, research has found that convolutional neural networks (CNN) have significant advantages in addressing computer vision problems. CNN is capable of automatically extracting image features from the deeper layers of the input images, eliminating the need for manual feature extraction, and performing quality regression on the extracted features. For example, Shi et al. (2020) proposed a multi-task CNN to simultaneously learn image quality assessment and distortion type recognition, extending a single-column multi-task CNN model to a three-column model, achieving good performance on asymmetrically distorted stereoscopic images. Chang, Li & Zhao (2020) proposed a hierarchical multi-scale no-reference SIQA method based on dilated convolution, which employed dilated convolution on newly generated monocular images to simulate multi-scale features of human vision. Li et al. (2020) also introduced a three-channel CNN-based SIQA method using three-dimensional visual saliency maps, which were derived from the two-dimensional saliency regions and depth saliency maps of stereo images. Bourbia, Karine & Chetouani (2021) proposed a novel network block called the Fusion Weight Allocation Module, which adaptively weighted features to guide the fusion of stereo image features. Sandić-Stanković, Kukolj & Le Callet (2022) utilized a general regression neural network to model the relationship between extracted features and subjective scores. While some deep learning-based NR-SIQA methods have achieved promising results in the literature, they often came with high training time complexity, resulting in lengthy processing times for generating results. Thus, accurately assessing the visual perceptual quality of stereoscopic images remained challenging.

Stereoscopic images in the real world often suffer from asymmetrical distortion, where the distortion level and type differ between left and right views. In order to solve the problem of inaccurate SIQA evaluation results caused by asymmetric distortion, researchers have made a series of efforts. Existing binocular perception studies suggest that during binocular integration in the HVS, both the right and left view images are simultaneously perceived, resulting in a fused image in the brain. However, during binocular rivalry in the HVS, only the right or left view is perceived. Therefore, the quality of stereoscopic images is not only related to the distortion level of each individual left or right view but also to the binocular stereoscopic perceptual experience. Inspired by this, this article proposes a monocular-binocular fusion-based SIQA (MB-FR-SIQA) method. The method proposed in this article has the following characteristics:

1. Traditional grayscale fusion views effectively address the issue of asymmetric distortion. We constructed fusion views from the red, green, and blue (RGB) color channels instead of the grayscale channel to ensure observers perceive the same spatial domain.

2. To ensure the integrity of feature extraction, we extracted local phase and local amplitude features based on extracting global features. We used saliency maps to optimize the quality score of local features.

3. When people observe images, both monocular and binocular visual characteristics occur simultaneously. This article extracted monocular and binocular features as quality-sensitive features and computed the final quality scores.

Build the RGB fusion view

We build RGB fusion views to simulate the brain's fusion of perceptual information from the left and right eyes. However, when there are varying degrees of distortion in stereoscopic image pairs, the visual stimuli received by the left and right eyes at the same retinal position are not equal, resulting in a mismatch in the perceived images and leading to the binocular rivalry phenomenon. Therefore, considering binocular rivalry is essential during the process of constructing fusion views. Given that Gabor filters can model frequency-oriented decomposition in the primary visual cortex and capture energy in a highly localized manner in both spatial and frequency domains (Field, 1987). We employ the energy response of Gabor filters to simulate visual stimulus intensity (Su, Bovik & Cormack, 2011), thus emulating binocular rivalry.

Constructed the fusion view on a grayscale channel in previous work (Messai, Hachouf & Seghir, 2020), and experimental results demonstrated its effectiveness for quality assessment. In this article, the fusion view is constructed on the red, green, and blue (R, G, B) channels to ensure that observers could see the same spatial domain. The RGB fusion view is defined as C(x, y)_n, where n ∈ {R, G, B}. The calculation formula of the RGB fused view is shown as follows:

(1) $$C(x,y{)}_n={\omega }_l(x,y{)}_n\times I_l(x,y{)}_n+{\omega }_r(x+m,y{)}_n\times I_r(x+m,y{)}_n$$where (x, y) represents the coordinates of the pixel. n is the color channel index. I_l(x, y) and I_r(x+m, y) represent the left and right views, respectively. m is the disparity index calculated using the SSIM-based stereo disparity algorithm. ω_l(x, y) and ω_r(x+m, y) respectively represent the weight coefficients of the left and right eyes. The weights are given by:

(2) $${\omega }_l(x,y)={\displaystyle \frac{G_l(x,y)}{G_l(x,y)+G_r(x+m,y)}}$$

(3) $${\omega }_r(x+m,y)={\displaystyle \frac{G_r(x+m,y)}{G_l(x,y)+G_r(x+m,y)}}$$where G_l(x, y) and G_r(x+m, y) are the total sums of amplitude responses of the Gabor filter bank in the left and right views, across eight orientations (horizontal, vertical, diagonal). These responses are used to simulate the visual stimulus intensity of the left and right views. We define the Gabor filter bank G(x, y) is calculated as follows:

(4) $$\begin{array}{rl}& G(x,y)={\displaystyle \frac{1}{2\pi {\sigma }_x{\sigma }_y}\cdot e^{-{\displaystyle \frac{1}{2}[{(R_1/{\sigma }_x)}^2+{(R_2/{\sigma }_y)}^2]\cdot e^{i(x{\zeta }_x+y{\zeta }_y)}}},}\\ & R_1=(x-m_x)\cdot \cos (\theta )+(y-m_y)\cdot \sin (\theta ),\\ & R_2=-(x-m_x)\cdot \sin (\theta )+(y-m_y)\cdot \cos (\theta )\end{array}$$where m_x and m_y determine the center of the Gabor filter's receptive field. σ_x and σ_y are the standard deviations of the elliptical Gaussian envelope along the x and y axes. ζ_x and ζ_y represent the spatial center frequencies of the complex sinusoidal grating. θ indicates the orientation of the filter.

The examples of synthesized RGB fusion views under symmetric and asymmetric distortion scenarios are shown in Fig. 2. As shown in Fig. 2A represents the undistorted reference left view, Fig. 2B represents the right view with Gaussian blur (BLUR) distortion, and Fig. 2C represents the RGB fusion view synthesized using the MB-FR-SIQA method. In Fig. 2C, the impact of noise is almost imperceptible. This is because Fig. 2A is undistorted, with higher contrast and clearer contours. Figure 2B is distorted, resulting in lower contrast and less distinct contours. Therefore, during the binocular competition process, Fig. 2A dominates the competition, leading to minimal noise in Fig. 2C. This demonstrates that the RGB fusion view can effectively reflect the binocular competition phenomenon. Panels Figs. 2D and 2E are examples of RGB fusion views synthesized from symmetrically JP2K distorted images using the MB-FR-SIQA method.

Extract binocular features

The RGB fusion view is generated through the complex process of binocular fusion and binocular competition in the human brain. Extracting effective feature information from the RGB fusion view can reflect the binocular quality. In this section, local features are first extracted, including local phase and local amplitude features. Then, the saliency map of the RGB fusion view is computed, which is used to adjust the weights of the local features. Finally, global features are extracted, including brightness, contrast, gradient magnitude, and phase consistency (PC) features.

Extract local features

Phase is an important feature in image quality assessment as it can capture a significant portion of the positional and structural information within an image. It exhibits high stability and adaptability and finds wide applications in image processing (Qian & Mikaelian, 2000). However, the global phase obtained through Fourier transform cannot effectively express local features. Log-Gabor filters can mimic the simple cell part of the HVS cortex in this article (Field, 1987). The log-Gabor filters maximize the fourier components to calculate the PC between the reference RGB fusion view and the distorted RGB fusion view. The formula for computing the energy response G_s,o(ω, θ) in the Fourier domain with four scales and four directions is given by:

(5) $$G_{s,o}(\omega ,\theta )=\exp \left[-{\displaystyle \frac{{(\log (\omega /{\omega }_s))}^2}{2{\sigma }_s^2}}\right]\times \left[-{\displaystyle \frac{{(\theta -{\theta }_o)}^2}{2{\sigma }_o^2}}\right]$$where ω is the radial coordinate. θ is the angular coordinate. s is the spatial scale index. o is the spatial orientation index. ω_s is the center frequency of the filter. θ_o is the orientation of the filter. Parameters σ_s and σ_o are used to control the strength of the filter.

A set of energy responses of the log-Gabor filter in the o direction and s scale is expressed as: [η_s,o(x), ξ_s,o(x)]. Then the local amplitude LA_s,o(x) at position x is calculated as follows:

(6) $$L{A}_{s,o}(x)=\sqrt{{\eta }_{s,o}{(x)}^2+{\xi }_{s,o}{(x)}^2}.$$

The local energy E_o(x) along the o direction at position x is calculated as follows:

(7) $$\begin{array}{rl}& E_o(x)=\sqrt{{\left(H_{\mathrm{o}}(x)\right)}^2+{\left(F_o(x)\right)}^2}\\ & H_o(x)=\sum _s{\eta }_{s,o}(x)\\ & F_o(x)=\sum _s{\xi }_{s,o}(x).\end{array}$$

The phase consistency PC_o(x) along the o direction at position x is calculated as follows:

(8) $$P{C}_o(x)={\displaystyle \frac{E_o(x)}{\sum _s{A}_{s,o}(x)+\epsilon }}$$where ε is a very small positive value. The global phase can not effectively express the local features. We use the local phase and local amplitude to describe the local features. The calculation formula for the local phase LP_local(x) at position x is defined as the angle along the direction of the maximum PC value between F_o(x) and H_o(x), and it is given by:

(9) $$L{P}_{local}(x)=\arctan (H_{o_m}(x),F_{o_m}(x))$$where o_m represents the direction corresponding to the maximum PC value. The local amplitude at point x is denoted as LA_local(x). It is defined as the sum of local amplitudes across all scales along the o_m direction. The calculation formula is shown as follows:

(10) $$L{A}_{local}(x)=\sum _s^{ }L{A}_{s,o_m}(x).$$

In this article, the log-Gabor filter is configured with four scales and four orientations. The parameter settings are as follows: ω_o = 1/6, θ_o = 0, σ_r = 0.3, σ_θ = 0.4. By using RGB fusion visualization, the resulting local phase and local amplitude images are shown in Fig. 3.

Saliency map

Different positions in an image contribute differently to the perception of image quality. To represent the visual importance of local regions, we use saliency maps. The saliency map is computed using an improved method called SDSP (Zhang, Gu & Li, 2013). Since the human eye is sensitive to changes in image edge positions, edge saliency is used instead of center saliency. First, people are more interested in warm colors. Second, bandpass filters can capture object features in visual scenes. Finally, human eyes are more sensitive to the perception of edge regions than smooth regions. The saliency map obtained by the method in this article clearly shows the shape and boundaries of objects. The saliency map is defined as VS(x), and the saliency map is calculated as follows:

(11) $$VS(x)=S_F(x)\cdot S_E(x)\cdot S_C(x)$$where S_F(x) is the saliency map at position x. S_E(x) is the edge saliency map at position x. S_C(x) is the color saliency map at position x. Based on the different types of distortions, the saliency maps generated through RGB fusion visualization are shown in Fig. 4.

Extract global features

The global features extracted based on the RGB fusion view include brightness, contrast, gradient magnitude, and PC. Among them, brightness and contrast represent the distortion of stereo image content, while gradient magnitude and PC represent the distortion of stereo image structure.

According to Wang et al. (2004), we define the brightness feature similarity at position x as S_L(x) and the contrast feature similarity as S_C(x). Their formulas for calculation are as follows:

(12) $$S_L(x)={\displaystyle \frac{2{\mu }_x{\mu }_y+C_1}{{\mu }_x^2+{\mu }_y^2+C_1}}$$

(13) $$S_C(x)={\displaystyle \frac{2{\sigma }_x{\sigma }_y+C_2}{{\sigma }_x^2+{\sigma }_y^2+C_2}}$$where, x and y represent the reference image and the distorted image, respectively. μ_x and μ_y represent the mean values of x and y. σ_x and σ_y represent the standard deviations of x and y. σ_xy represents the covariance between x and y. C₁ and C₂ are constants used to avoid errors caused by division by zero in the denominators.

The gradient magnitude is an effective way to capture the structural features of an image. We use the Prewitt operator (Nezhadarya & Ward, 2009) to obtain the gradient map of the RGB fusion view. The gradient magnitude represents the structural information of the stereo image. The gradient magnitude GM(x) at position x is calculated as follows:

(14) $$\begin{array}{rl}& GM(x)=\sqrt{G_x{(x)}^2+G_y{(x)}^2}\\ & G_x(x)={\displaystyle \frac{1}{3}\left[\begin{array}{ccc}1& 0& -1\\ 1& 0& -1\\ 1& 0& -1\end{array}\right]\otimes f(x)}\\ & G_y(x)={\displaystyle \frac{1}{3}\left[\begin{array}{ccc}1& 1& 1\\ 0& 0& 0\\ -1& -1& -1\end{array}\right]\otimes f(x)}\end{array}$$where ⊗ represents the convolution operation. f(x) represents the image. G_x(x) and G_y(x) represent the horizontal and vertical gradients.

The formula for calculating the similarity of gradient magnitudes S_GM(x) at position x is shown as follows:

(15) $$S_{GM}(x)={\displaystyle \frac{2G{M}_{ref}(x)\times G{M}_{dis}(x)}{G{M}_{ref}{(x)}^2+G{M}_{dis}{(x)}^2+C_3}}$$where C₃ is a constant used to prevent division by zero. GM_ref(x) and GM_dis(x) represent the gradient magnitude of the RGB reference and distorted fusion views.

PC provides relative invariance to image variations and facilitates the extraction of stable features in the image. The calculation formula of the similarity S_PC(x) of PC at position x is shown as follows:

(16) $$S_{PC}(x)={\displaystyle \frac{2P{C}_{ref}(x)\times P{C}_{dis}(x)}{P{C}_{ref}{(x)}^2+P{C}_{dis}{(x)}^2+C_4}}$$where C₄ is a constant used to prevent division by zero. PC_ref(x) and PC_dis(x) represent the PC of the RGB reference and distorted fusion views.

Extract monocular features

In stereo images, the feature information from the left and right views can effectively reflect the quality of the stereo image. According to Formulas (12), (13), and (15), the brightness similarity, contrast similarity, and gradient magnitude similarity of the left and right views are calculated respectively. Then effectively fuse the above three feature similarities. Taking the left view as an example, the formula for calculating the similarity S_l(x) between the reference left view and the distorted left view at position x is shown as follows:

(17) $$S_l(x)={\left[S_L(x)\right]}^{\alpha }\cdot {\left[S_C(x)\right]}^{\beta }\cdot {\left[S_{GM}(x)\right]}^{\gamma }$$where α, β, and γ are the balance parameters for adjusting brightness, contrast, and gradient magnitude, respectively. We consider all three factors to be equally important. Set α = β = γ = 1. Similarly, we calculate the feature similarity S_r(x) for the right view.

Quality assessment based on monocular and binocular features

When observing stereo images, both monocular and binocular vision phenomena occur simultaneously. Therefore, fusing monocular and binocular features improves the predictive ability of image quality. We calculate monocular and binocular quality scores based on their respective features. And we propose a merging method to combine them into an overall quality score for stereo images. The quality assessment process is divided into four steps, which are detailed as follows.

The first step is to extract global and local features based on the RGB fusion view, which effectively represents the binocular quality. For the local phase and local amplitude features, similarity can be computed using a method similar to Eq. (15). Then the local phase feature similarity S_LP(x) and local amplitude feature similarity S_LA(x) are calculated as follows:

(18) $$S_{LP}(x)={\displaystyle \frac{2L{P}_{local}^{ref}(x)\cdot L{P}_{local}^{dis}(x)+{\epsilon }_1}{{\left(L{P}_{local}^{ref}(x)\right)}^2+{\left(L{P}_{local}^{dis}(x)\right)}^2+{\epsilon }_1}}$$

(19) $$S_{LA}(x)={\displaystyle \frac{2L{A}_{local}^{ref}(x)\cdot L{A}_{local}^{dis}(x)+{\epsilon }_2}{{\left(L{A}_{local}^{ref}(x)\right)}^2+{\left(L{A}_{local}^{dis}(x)\right)}^2+{\epsilon }_2}}$$where ε₁ and ε₂ are constants used to prevent division by zero. $L{P}_{local}^{ref}(x)\mathrm{a}\mathrm{n}\mathrm{d}L{A}_{local}^{ref}(x)$ represent the local phase and local amplitude features extracted from the reference RGB fusion view. $L{P}_{local}^{dis}(x)\mathrm{a}\mathrm{n}\mathrm{d}L{A}_{local}^{dis}(x)$ represent the local phase and local amplitude features extracted from the distorted RGB fusion view. To comprehensively combine the local features, we integrate the local phase and local amplitude to form an overall local quality score S₁(x), which is given by:

(20) $$S_1^{ }(x)=W_{LP}\cdot S_{LP}(x)+W_{LA}\cdot S_{LA}(x)$$where W_LP is the weight for the local phase. W_LA is the weight for the local amplitude. The experimental results show that in the LIVE 3D Phase I database (LIVE P-I), W_LP = 0.6 and W_LA = 0.4 yield the best results. In the LIVE 3D Phase II (LIVE P-II) database, W_LP = 0.75 and W_LA = 0.25 yield the best results.

For a given position x, if either the reference or distorted RGB fusion view has a high saliency value, then position x has a high impact on the HVS when evaluating the similarity between the two images (Henriksson, Hyvärinen & Vanni, 2009). Therefore, we weight S₁(x) with higher saliency values in the reference and warped RGB-fused views to obtain an optimized binocular local feature quality score Q₁, calculated as follows:

(21) $$\begin{array}{rl}& Q_1={\displaystyle \frac{\sum { }_{x\in \mathrm{\Omega }}{S}_1(x)\cdot V{S}_m(x)\cdot S_{VS}(x)}{\sum { }_{x\in \mathrm{\Omega }}V{S}_m^{ }(x)}}\\ & S_{VS}(x)={\displaystyle \frac{2V{S}_{\mathrm{r}\mathrm{e}\mathrm{f}}(x)\cdot V{S}_{\mathrm{d}\mathrm{i}\mathrm{s}}(x)+{\epsilon }_3}{V{S}_{\mathrm{r}\mathrm{e}\mathrm{f}}^2(x)+V{S}_{\mathrm{d}\mathrm{i}\mathrm{s}}^2(x)+{\epsilon }_3}}\\ & V{S}_m(x)=max(V{S}_1(x),V{S}_2(x))\end{array}$$where Ω is the spatial domain of the image. VS_m(x) represents the higher saliency value between the reference and distorted RGB fusion views. S_VS(x) represents the feature similarity of the saliency map. VS_ref(x) and VS_dis(x) are the saliency maps of the reference and distorted RGB-fused views. ε₃ is a constant to prevent division by zero.

In the second step, global features are extracted based on the RGB fusion view, including brightness, contrast, gradient magnitude, and PC features. The gradient magnitude similarity is calculated using Eq. (15), and the PC similarity is calculated using Eq. (16). Then, the gradient magnitude and PC are effectively fused, representing the structural similarity S₂(x). Calculated as follows:

(22) $$S_2(x)=S_{PC}(x)\cdot S_{GM}(x)$$

To obtain accurate structural similarity, we use the larger PC values from the reference and distorted RGB fusion views to weigh the overall structural similarity:

(23) $$\begin{array}{rl}& S_3(x)={\displaystyle \frac{\sum { }_{x\in \mathrm{\Omega }}{S}_2(x)\cdot P{C}_m(x)}{\sum { }_{x\in \mathrm{\Omega }}P{C}_m(x)}}\\ & P{C}_m(x)=max(P{C}_{ref}(x),P{C}_{dis}(x))\end{array}$$where Ω is the spatial domain of the image. PC_m(x) indicates higher PC values in reference and distorted RGB fused views.

Calculate the structural similarity S₃(x) according to Eq. (23). Calculate brightness similarity S_L(x) and contrast similarity S_C(x) according to Eqs. (12) and (13). The MS-SSIM (Wang, Simoncelli & Bovik, 2003) method is used to calculate the binocular quality score Q₂ for the three types of similarity information:

(24) $$Q_2=\sum _{x\in C}{\displaystyle \frac{Q_{MS-SSIM}(C_{ref}(x),C_{dis}(x))}{N_C}}$$where N_C represents the number of pixels in the RGB fused view. C_ref(x) and C_dis(x) represent the original fused view and the distorted fused view.

In the third step, we calculate the quality score S_l(x) of the left view according to Eq. (17). And we calculate the quality score S_r(x) of the right view in the same way. Then the calculation of monocular quality score Q₃ is given by:

(25) $$Q_3={\displaystyle \frac{S_l(x)+S_r(x)}{2}}$$

In the fourth step, the final quality prediction score Q is calculated by fusing the monocular and binocular quality scores. The overall quality score Q is calculated as follows:

(26) $$Q=a{Q}_1^{ }+b{Q}_2^{ }+c{Q}_3$$where a, b, and c are different weights assigned to Q₁, Q₂, and Q₃, and a + b + c = 1. During the experimental process, different parameter combinations are used to achieve the best results. In the LIVE P-I, the best-predicted quality score is obtained when a = 0.5, b = 0.2, and c = 0.3. In the LIVE P-II, the best-predicted quality score is obtained when a = 0.6, b = 0.2, and c = 0.2.

Results and analysis

This section provides a comprehensive assessment of the performance of the method. Firstly, it describes the databases and performance metrics used in the experiments. Secondly, it presents a comparative analysis of the overall performance of the method against other relevant methods, as well as performance comparisons for each type of distortion. Then perform consistency performance prediction and ablation experiments and time complexity analysis.

SIQA databases

We evaluate the effectiveness and robustness of the MB-FR-SIQA method on two publicly vailable databases (Ye et al., 2012).

LIVE P-I database consists of 20 reference stereo image pairs and 365 symmetrically distorted stereo image pairs. All stereo image pairs exhibit symmetric distortions, meaning that the left and right views have the same types and degrees of distortion. The distortion types include JPEG, JPEG2000 (JP2K), White Noise (WN), Gaussian Blur (BLUR), and Fast Fading (FF). There are 45 image pairs with WN distortion, and each of the other distortion types has 80 image pairs.

LIVE P-II database contains both symmetric and asymmetrically distorted stereo image pairs. It comprises a total of eight reference image pairs and 360 distorted image pairs. The distortion types are the same as those in the LIVE P-I database. Each distortion type contains 72 image pairs. The subjective quality scores in both databases are represented by Differential DMOS. The smaller the DMOS value, the smaller the difference and the better the quality. Figure 5 shows the left and right views of different distortion types in the LIVE P-I database.

Performance assessment index

To effectively evaluate the performance of the MB-FR-SIQA method. We adopt three commonly used assessment metrics to evaluate its performance: Pearson Linear Correlation Coefficient (PLCC), Spearman Rank Order Correlation Coefficient (SROCC), and Root Mean Square Error (RMSE). PLCC assesses the accuracy of the method. SROCC judges the monotonicity of the method and RMSE reflects the consistency of the method. The closer PLCC and SROCC are to 1, and the closer RMSE is to 0, the better the prediction performance of the method. For nonlinear regression between subjective and objective scores, we use the five-parameter logistic function (Sheikh, Sabir & Bovik, 2006) to compute:

(27) $$f(x)={\beta }_1\times \left[{\displaystyle \frac{1}{2}-{\displaystyle \frac{1}{1+\exp ({\beta }_2(x-{\beta }_3))}}}\right]+{\beta }_{4}x+{\beta }_5$$

Among them, x represents the predicted score. f(x) is the mapped score. β₁, β₂, β₃, β₄, β₅ are parameters determined by the best fit between subjective and objective scores.

Method performance

The experiments are conducted in two stages on the LIVE 3D IQA database to validate the effectiveness and consistency of the proposed BM-FR-SIQA method in this article. Firstly, the performance of local feature extraction, overall performance evaluation, and performance evaluation for a single type of distortion is being verified. Then, consistency performance prediction and ablation experiments are being conducted, followed by a summary of this chapter.

Local feature extraction performance

Feature extraction is one of the most critical procedures in SIQA. Better feature extraction results not only help to improve the accuracy of the SIQA method but also help to coincide with subjective results. Our experiment uses four different schemes in the LIVE P-I and LIVE P-II databases to demonstrate the effectiveness of feature extraction. All schemes are based on RGB fused views, and their basic composition is as follows: Scheme 1 only uses local phases to calculate local feature scores. Scheme 2 uses only local amplitudes to compute local mass fractions. Scheme 3 uses a combination of local phase and amplitude to calculate the mass fraction. Scheme 4 is the MB-FR-SIQA method, which uses the saliency map to weigh the local phase and amplitude to calculate the quality score. The overall performance comparison results of the four schemes and the MB-FR-SIQA method are presented in Table 1. Compared with Schemes 1 and 2, Scheme 3 uses a single local phase or local amplitude which is not as effective as fusing local phase and amplitude. Compared with Scheme 3, Scheme 4 is effective in using saliency-weighted local features.

Overall performance assessment

In the LIVE P-I database, all images are symmetric, whereas 66% of the images in the LIVE P-II database exhibit asymmetric distortions. To investigate the performance of the proposed method on symmetric and asymmetric distortion databases, experiments are conducted in two phases of the LIVE 3D IQA database in this article. To comprehensively evaluate the performance of the MB-FR-SIQA method, we compare the performance of the MB-FR-SIQA method with 10 existing SIQA methods including three NR-SIQA methods and seven FR-SIQA methods.

Table 2 shows the overall performance of the different approaches in the above two databases. In Table 2, it can be observed that in both databases, traditional methods like SSIM (Wang et al., 2004) and MS-SSIM (Wang, Simoncelli & Bovik, 2003) perform poorly compared to the MB-FR-SIQA method due to their insufficient consideration of stereo characteristics. Benoit, Le Callet & Campisi (2008) heavily depended on stereo matching. Chen et al. (2013) constructed fused views from reference and distorted stereo image pairs and then used 2D-IQA methods to evaluate stereo image quality. However, due to the differences between 2D and 3D images, the assessment results may not be accurate. Liu, Kong & Zhen (2019) improved the predictive performance of SIQA to some extent by considering the binocular interaction process of the HVS, but they ignored the influence of features from the left and right views on quality assessment. Shao et al. (2013) considered both global and local features, but some other important HVS features were neglected. Among NR methods, Messai, Hachouf & Seghir (2020) also obtained fused views and disparity maps from stereo image pairs, but only extracted gradient features related to them. They neglected other important visual features. Due to the superiority of CNN networks in feature extraction, Oh et al. (2017) performed better than other NR methods in the Phase I database, but it failed to achieve the expected performance in the Phase II database due to the lack of stereo features.

Overall, the method proposed in this article achieves optimal performance in terms of PLCC and RMSE on the LIVE P-I database, with SROCC also ranking among the top. On the LIVE P-II database, the addition of asymmetric distortions to stereo images introduces binocular rivalry during observation. However, the RGB fusion view employed in this article effectively mitigates the binocular rivalry caused by asymmetric distortions. Consequently, the performance metrics of the MB-FR-SIQA method on the LIVE P-II database surpass those of other methods.

Single distortion type performance assessment

An effective SIQA method should not only demonstrate overall accuracy across the entire database but also be capable of predicting quality for individual distortion types. In this section, we conduct separate experiments for each distortion type in the LIVE P-I and LIVE P-II databases to validate the generality of the proposed method. The PLCC and SROCC values for each distortion type are in different databases in Tables 3 and 4, including JP2K, JPEG, WN, FF, and BLUR. The MB-FR-SIQA method exhibits the best performance in predicting JP2K and WN distortion types in the Phase I database, while in the Phase II database, it performes best in predicting JP2K and BLUR distortion types. For other distortion types, the MB-FR-SIQA method maintains competitive performance, showing certain advantages over the comparative methods.

Consistency performance prediction

To provide a more comprehensive analysis of the experimental results, we visualize the overall differences between subjective assessment scores and predicted scores. The scatter plots of the predicted values and DMOS scores for the MB-FR-SIQA method on the LIVE P-I and LIVE P-II databases in Fig. 6. The x-axis represents DMOS scores, while the y-axis represents the predicted quality scores of the distorted images. The closer the scatter points are to the fitting curve, the higher the consistency between the predicted values and subjective assessment scores. Based on the distribution of scatter points in the figure, it can be observed that the subjective and objective scores tend to be close, indicating that the MB-FR-SIQA method demonstrates good subjective consistency.

Ablation experiment

We use ablation experiments to verify the effectiveness of the MB-FR-SIQA method. Plan A involves extracting monocular features only for quality score calculation, Plan B trains by solely utilizing binocular features for quality score calculation, Plan C calculates quality scores by fusing both monocular and binocular features. Table 5 presents the comparison of results under different plans. The best-performing is displayed in the bold. Experimental results indicate that combining monocular and binocular visual features can provide a more reasonable and accurate quality assessment method.

Time complexity analysis

Generally, time complexity is used to describe the running time and efficiency of an algorithm. In order to prove the advantages of this method in terms of time complexity, a comparative experiment is conducted in this section. The experimental environment is: Windows10 system, 11th Gen Inter(R) Core(TM) i5-11400H @ 2.70GHz processor, 16GB RAM, Matlab 2018b. Table 6 shows the average quality score calculation time of each stereo image pair for different methods on LIVE P-I. As can be seen from the table, compared with the methods proposed by Messai, Hachouf & Seghir (2020) and Lin & Wu (2014), our method has lower time complexity, which shows that our method has excellent performance in terms of running speed. For the method proposed by Messai, Hachouf & Seghir (2020), it takes the longest time because it uses machine learning methods for training and testing. For the method proposed by Lin & Wu (2014), the feature extraction process is very complex, so the calculation time is doubled compared to the method in this article. Although the method proposed by Liu, Kong & Zhen (2019) has a shorter runtime, its performance is lower than that of the method presented in this article due to the neglect of monocular features.