New media art design based on fast visual segmentation and 3D image processing

Improved U-net

Figure 2 depicts the improved U-Net network’s organizational structure. Set the U-Net network input as a single channel, with a classification number of 5, to meet the requirements of identifying art images and unify all training art images to 256 × 256 resolution, adding spatial attention and parity cross convolution. Training a deep learning model on high-resolution images can be computationally intensive and time-consuming. The computational burden is reduced by resizing all training images to a standardized resolution, such as 256 × 256, enabling faster iterations during training. Spatial attention and parity cross-convolution can effectively enhance the features of art images and expand the perceptual field, thus improving the segmentation accuracy of the network. The setting of multiple convolution layers can help the network fully extract and identify features.

Spatial attention

In visual segmentation tasks, good image features can significantly improve the segmentation accuracy of the algorithm. Convolutional neural networks have long been the mainstream selection approach for image segmentation. From the development history of convolutional neural networks, good graph features largely determine the accuracy of image segmentation tasks. The feature maps in convolutional neural networks are usually segmented into different regions. Each region of the feature map has a very different contribution to the result for a particular image segmentation task. Therefore, it is necessary to increase the number of features extracted from the target region by the network and to suppress invalid or inefficient features to obtain more beneficial feature information for a particular visual task. Spatial attention16 is an optimization scheme designed to simulate human thinking, and its core idea is to redistribute resources based on contribution values, which can improve the attention of neural networks to target regions. Since the study aims to segment various elements in the art image, spatial attention is introduced to change the weight values of different feature regions of the image. The feature regions with high relevance will have higher weights so that the target regions needed for the task will be more prominent, helping the network to acquire more features and achieve better segmentation results. Figure 3 shows its structure.

The specific implementation of the spatial attention is to max pooling and average pooling of the feedback characteristic graph to get the results Xmax and then splice the results to get the composite signal Xconcat. 1 × 1 convolution is used to one-dimensional the signal to obtain the weight graph. The final output signal is the input signal affected by the weight chart, as shown in Formula (1) and Formula (2). (1)${\displaystyle Xconcat=Concat\left(Xmax+Xavg\right)}$ (2)${\displaystyle Xout=Xin\cdot \delta \left[conv\left(Xconcat\right)\right]}$

Formulas (1) and (2) splice the results to obtain composite signals Xconcat. 1 × 1 convolution is used to one-dimensional the signal to obtain the weight graph. The final output signal is the input signal affected by the weight chart.

Parity cross convolution

Within image segmentation, the perspicacious delineation of objects stands as the focal point and crux of this endeavor. In art imagery, the inherent characteristics often lack the conspicuousness requisite for facile recognition, thereby conferring an inherent complexity to the segmentation process. In the paradigm of U-net architecture, as one delves deeper into the strata of network layers, the convolutional layers bequeath semantic information of substantial potency, thus furnishing a promising foundation for the precision of target segmentation. However, this advantage is juxtaposed with a miniature perceptual field that inadvertently elides myriad intricate details.

Conversely, the shallower convolutional layers yield a more expansive perceptual field, harboring a profusion of intricate target particulars that bear relevance to the task of specific segmentation. Regrettably, a paucity of semantic import undercuts this detail bounty and impairs intricate segmentation objectives. Compounding this predicament is the quandary emanating from the augmented resolution of features gleaned from deeper network strata. In contrast, it seemingly augments potential. It frequently constricts the target’s mapping scope to select diminutive and deep-seated feature maps, consequently impinging upon the targeted segmentation’s exactitude.

The singular integration of spatial attention into the U-net architecture, although beneficial in accentuating feature relevance, regrettably neglects the amplification of the perceptual field. Moreover, U-net’s utilization of stepwise odd convolution for its downsampling stratum engenders a notably skewed distribution of the perceptual domain, thereby stifling the network’s capacity to distill granular semantic information and consequently imperiling the accuracy of segmentation.

To ameliorate these complexities, this article introduces an innovative convolutional framework that synergistically melds even and odd convolutional layers, thereby engendering a parity cross-convolution schema. The efficacy of this integration is epitomized as follows: (3)${\displaystyle Y=F3\left(\sigma \left(F2\left(X\right)\right)\right)}$ X, Y indicates the input and output, respectively. F3 and F2 denote the 3 × 3 and 2 × 2 convolution layers, respectively. σ is the nonlinear operation.

To reduce the complexity further, a 2 × 2 even convolution layer is constructed before the U-net convolution layer in this article to realize the connection adjustment for the subsequent downsampling. Figure 4 shows the stepwise odd convolution and the parity cross convolution. The convolution of 2 × 2 has the lowest computational complexity compared to other convolution layers, so it can minimize the time complexity based on the effective elimination of perceptual field inhomogeneity. In the parity cross convolution, the padding parameters of the convolution are set to 1 and 0, respectively. With an even number of inputs, this padding eliminates the output asymmetry caused by the stepwise odd convolution.

The perceptual field distribution after using the parity cross-convolution is given in Fig. 5, where the input image is 6 × 6 and the feature map is 3 × 3. After using the parity cross convolution, the perceptual field intensity distribution of each point of the input tends to be homogeneous, which can effectively avoid the systematic bias caused by the stepwise odd convolution, thus making full use of each pixel and finally improving the segmentation accuracy of the image semantics.

Loss function

The overall loss function used for the network in this article is: (4)${\displaystyle L=\lambda 1Lseg+\lambda 2Ledge}$ where Lseg is the weighted loss (Zhou et al., 2018). The Formula is expressed as: (5)${\displaystyle Lseg=-\sum _{c=1}^{M}wcyclog\left(pc\right)}$ (6)${\displaystyle wc=\frac{N-Nc}{N}}$ N is the total number of pixels. Nc is the pixel of category c.yc isa vector with elements taking only 1 and 0. 1 is taken if the category is the same, otherwise 0 is taken. pc indicates the probability that the predicted sample belongs to c.

Ledge is the binarized cross-entropy loss function, defined as follows (Tan et al., 2021). (7)${\displaystyle Ledge=-\sum _i\left(bilog\hat{b}i+\left(1-bi\right)log\left(1-\hat{b}i\right)\right)}$ bi and $\hat{b}i$ is the annotated image pixel values and the edge prediction feature image pixel values, respectively.

Iterative training of the network leads to an inconsistent number of results for Lseg and Ledge, so the metric scales are made consistent by weighting the coefficients. The coefficients λ1 and λ2 are set to 1 and 10, respectively.

3D reconstruction of artistic image based on isosurface extraction algorithm

This article reconstructs the segmented art image data in 3D images to get the 3D model of art images. Conventional reconstruction methods mainly include volume rendering and surface rendering. The surface rendering method has the advantages of low resource consumption and fast running speed. This article uses the classical surface rendering reconstruction algorithm-the isosurface extraction algorithm (Zhao et al., 2022).

As shown in Fig. 4, the central idea of the isosurface extraction algorithm is The main idea of the algorithm is to detect the voxel elements that intersect with the isosurface, calculate the coordinates of the intersection points, and then construct the corresponding grid topology for different intersections. A voxel created by eight pixels from two adjacent slices is used to locate the surface. In this article, each grid cell of the data set composed of 3D data is regarded as a voxel. For the convenience of processing, it is specified that each voxel has eight vertices. The vertex values greater than the isosurface are identified as positive; otherwise, they are. Therefore, building a table can accurately reflect 256 kinds of relationships between a voxel and the isosurface. Unifying training images to a resolution of 256 × 256 is a practical strategy that balances computational efficiency, memory constraints, model architecture considerations, and the trade-off between detail and context. It contributes to a stable and efficient training process, promoting generalization and optimal segmentation performance.

Since the relationship between the vertex and the isosurface is entirely symmetrical, we can reduce all cases to 128. The cube is rotationally symmetric, which can be reduced to 14 cases, and then the edge of each voxel element is indexed, as shown in Fig. 6.

The same method is used to record the relationship of edges, and then the intersection points are confirmed by linear interpolation to generate local triangles. Then, connect all the triangles to get the reconstructed 3D model. “Local triangles” refer to small triangular facets that collectively form a mesh representation of the reconstructed object’s surface. Each local triangle consists of three vertices in 3D space and represents a planar segment of the object’s surface. These local triangles are crucial building blocks for constructing a complete 3D model. Isosurface images are vital in visualizing and representing the reconstructed 3D object. An isosurface image portrays a specific value (known as an iso-value) within a 3D volume dataset. In 3D reconstruction, the volume dataset is often derived from the point cloud or the voxel representation obtained during reconstruction. To create an isosurface image, a threshold value is chosen. All points or voxels with values more significant than this threshold are considered part of the object’s surface. The isosurface image is then generated by connecting these surface points, effectively forming a 2D representation of the object’s surface at the chosen iso-value. This image visually depicts the reconstructed object’s shape, facilitating analysis and interpretation.

Specifically, there are seven steps as follows:

1. Input segmentation results.

2. Create voxels in order.

3. Calculate the index by comparing the vertex size of the voxel and isosurface;

4. Use the index to find the corresponding parameter from the table.

5. Repeat step 3 to calculate the edge.

6. Generate local triangles according to vertex and edge conditions.

7. Combining triangles and drawing isosurface images.

Training model

The in-depth learning platform of this experiment is Python 1.7.0. NVIDIA GTX2080ti GPU is used for training. Five thousand photos are selected from VOC2007 to train the improved U-net network. The training curve is shown in Fig. 7. The images are converted from RGB format to grayscale images to facilitate the training. Optimized training using the RMSprop algorithm.

The RMSprop algorithm is improved based on the AdaGrad algorithm (Li, Chen & Zhang, 2019). The principle determines a global learning rate, and the global learning rate is continuously divided by the square root of the sum of squares of the historical gradients controlled by the decay coefficient in each learning. The RMSprop algorithm formula is as follows. (8)${\displaystyle Sdw=\beta Sdw+\left(1-\beta \right)d{w}^2}$ (9)${\displaystyle Sdb=\beta Sdb+\left(1-\beta \right)d{b}^2}$ (10)${\displaystyle w=w-\alpha \frac{dw}{\sqrt{Sdw+\varepsilon }}}$ (11)${\displaystyle b=b-\alpha \frac{db}{\sqrt{Sdb+\varepsilon }}.}$

Among them, Sdw and Sdb represent the gradient momentum of weight w and offset value b in iteration, α represent the learning rate, β represent the super parameter, and β represent non-zero number.

After training, the art image is segmented, and then the input segmentation results are reconstructed according to the isosurface extraction algorithm to obtain the 3D model of the art image.

Experimental results of image segmentation

One hundred images were predicted using the training model and then the dataset was segmented and compared to evaluate the training model and obtain its mean intersection over union (MIoU) parameters (He, Fang & Plaza, 2020). The MIoU parameters of the conventional U-net network and its variants and the improved U-net network in this article are shown in Table 1.

The larger the MIoU value, the better the segmentation of the target. It can be seen that the algorithm in this article has achieved the highest MIoU parameters. Compared with the original U-net, the segmentation effect on elements is 6.4% higher, the segmentation effect on the background is 0.2% higher, and the overall segmentation effect is 3.3% higher. This is mainly because the attention mechanism and parity cross-convolution added in this article can extract more detailed features. The quantitative analysis of MIoU parameters shows that the improved algorithm has better segmentation ability.

Comparison of different algorithms

The segmentation performance of the method used in this article needs to be further validated. It is compared with FCN14, SegNet16 ACU-Net (Re, Stanczyk & Mehrkanoon, 2021), ATU-Net (Wang, Li & Zhuang, 2021) and ENet (Zhou et al., 2018). The dataset of self-created art images served as the testing dataset for all experiments in this article, and the software and hardware environment and parameter settings of the investigation were the same. Table 2 gives the comparison results.

Examining the findings presented in Table 2, it becomes evident that the methodology employed in this study leads to exceptional outcomes in terms of both Mean Intersection over Union (MioU) and the number of parameters utilized. Remarkably, our approach emerges as the top performer, reflecting its prowess in achieving optimal segmentation accuracy while maintaining efficiency in parameter utilization.

Considering the frames per second (FPS) metric, which serves as a gauge for real-time performance and lightweight network representation, ENet emerges as the frontrunner, delivering commendable results in this aspect. Notably, it outperforms other methods, such as FCN, which ranks at the lower end of the FPS spectrum. This juxtaposition underscores ENet’s capacity for swift real-time processing.

Our algorithm substantially enhances real-time performance compared to competing models like FCN, ACU-Net (Re, Stanczyk & Mehrkanoon, 2021), ATU-Net (Wang, Li & Zhuang, 2021), and SegNet. While the segmentation speed might not match ENet’s, our approach secures a noteworthy advantage in terms of MioU, exhibiting superior segmentation accuracy and fidelity. This duality of attributes showcases our method’s ability to balance segmentation precision and operational speed, thereby underscoring its suitability for various practical applications.

3D reconstruction experiment

To objectively reflect the image reconstruction performance of the method in this article, peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are selected as the image reconstruction quality evaluation indicators (He, Fang & Plaza, 2020). Select ten images from the art image data set, conduct image reconstruction quality comparison experiments with conventional methods, AWAN16 and HRNet (Tan et al., 2021), and evaluate the image reconstruction quality with selected evaluation indicators. The average reconstruction quality data of 10 images are shown in Table 3.

The results presented in Table 3 underscore the remarkable performance of the method proposed in this article. Our approach attains the most impressive PSNR and SSIM values, standing at 38.16 and 0.9808, respectively. Remarkably, the average time required to reconstruct a single image is 1.2 s. This efficiency significantly outshines the performance of the alternative three methods under comparison. Consequently, it becomes evident that our method excels in achieving both precision and expeditiousness in the reconstruction of art images, a testament to its efficacy and practicality in the domain.