Visual resource extraction and artistic communication model design based on improved CycleGAN algorithm

Image style migration

Image style migration is an innovative technique designed to transmute the style of a given image into that of another designated content image. This approach offers the flexibility to apply diverse artistic styles to images, enabling the application of landscape-style paintings to ordinary photographs, thereby realizing a transformative filtering function. The origins of image-style migration techniques can be traced back to the 1990s when the focus gradually shifted towards their development. Most texture migration algorithms employ non-parametric methods to synthesize image textures while simultaneously preserving the image’s underlying structure, utilizing a variety of distinct techniques. Efros & Freeman (2001) pioneered the incorporation of texture synthesis concepts to generate stylized images. Frigo et al. (2016) introduced an instance-based partitioning approach that varied patch sizes and sought to identify the optimal patch matches between the target image and the reference image. Elad & Milanfar (2017) proposed a classical patch-matching-based style migration technique with adaptive patch enhancements aimed at augmenting image quality. Traditional style migration algorithms adeptly simulate specific artistic styles but face limitations when extending to other, unexplored styles. For novel styles, a considerable investment of time is required to analyze the nuances of the style, drawing upon substantial human knowledge and experience. Furthermore, traditional style migration approaches predominantly extract the foundational features of an image. Consequently, the primary drawback of conventional style migration techniques lies in their inability to produce satisfactory stylized images when dealing with images characterized by intricate color palettes and complex textures Before the advent of deep learning, the methodologies employed for image style migration were primarily centered around image rendering. These image rendering techniques are often categorized into stroke rendering, region rendering, and instance rendering, relying on image processing filter techniques (Hertzmann et al., 2001). Although these rendering methods effectively bestow style upon content images, they predominantly operate on the image’s underlying information, often failing to encapsulate the abstract features that genuinely encapsulate the content’s essence.

The ascent of CNN models within the domain of image processing has been characterized by remarkable milestones, commencing with AlexNet in 2012 and culminating with VGGNet in 2014 and ResNet in 2015. As the CNN model matures post-2016, style migration techniques increasingly adopt deep learning to facilitate the transmutation of image styles. Gatys, Ecker & Bethge (2015), in their groundbreaking study, discerned the feasibility of segregating image content and style representations across different layers of CNN. Building upon this insight, they advanced an image iteration-based style migration algorithm (Gatys, Ecker & Bethge, 2016). Johnson, Alahi & Fei-Fei (2016) on the other hand, introduced a swift image style migration technique, wherein the desired stylized image is generated via the construction of an image generation network. These endeavors collectively underscore the evolving landscape of style migration, marked by the growing integration of diverse neural network methodologies aligned with the burgeoning capabilities of deep learning technology. Consequently, the expansion of research applications in the realm of style migration through deep learning networks has surged to the forefront of contemporary research endeavors, affirming its status as a compelling and vibrant area of exploration in recent years.

Image migration method based on generative adversarial networks

GANs ascertain the fidelity of generated samples by learning the underlying data distribution, thus addressing the challenge of not having exact sample matches. Instead of generating an average outcome from all reasonable samples, GANs strive to generate a more plausible sample within the generative space. In 2014, Goodfellow et al. (2014) introduced the GAN model, comprising a generator (G) and a discriminator (D). GANs optimize both the generator and discriminator through adversarial training, whereby the generator endeavors to produce the most convincing results. Radford, Metz & Chintala (2017) proposed DCGANs, which integrate GANs with deep CNN architectures and are tailored for specialized image generation. Chen, Lai & Liu (2018) presented CartoonGAN, designed to produce cartoon-style images by incorporating edge facilitation loss. This technique can generate diverse cartoon images when provided with a real scene as the content image. Isola et al. (2017) introduced CGAN-based image-to-image translation, requiring paired data during the training process. Zhu et al. (2017) developed CycleGAN, a model that eliminates the need for paired images and enables image-to-image conversion. Hu, Ding & Li (2020) devised a style migration model based on GAN, abandoning the redundant structure of two GAN models trained by CycleGAN. Instead, they employ feature maps from the VGG network to constrain the semantic content between the input image and the generated image. This approach streamlines training, with only one generator and one discriminator being employed, thereby saving time. Style migration plays a pivotal role in various fields, including art, film and television post-production, and game scene development. CycleGAN, DiscoGAN, and DualGAN, all rooted in the concept of dyadic learning, are parallel efforts in the realm of GAN (Yi et al., 2017). These models enable style migration tasks with just the dataset, transcending the need for image pairing and significantly expanding the application scope of style migration. Choi et al. (2018) proposed the StarGAN model, which can achieve the transfer of multiple styles within a single GAN. Although the StraGAN model can achieve multiple styles of transfer, the number of styles is still limited, and cannot achieve the transfer of any style. Xu et al. (2020) combined the AdaIN module with the generative adversarial network and added the AdaIN module to the adversarial network, enabling the network to generate images of any style. Cho et al. (2019) combined WCT with GAN and proposed the GDWCT model, which approximates WCT by using regularization and grouping calculations while reducing the number of parameters.

The studies outlined above vividly illustrate the substantial progress made in the domain of deep learning-based generative content research, particularly concerning images. Deep learning-based style migration techniques have exhibited impressive advancements, significantly enhancing migration speed and migration quality. While various GAN networks have seen improvements, their fundamental frameworks have remained relatively consistent (Kim et al., 2017). Consequently, in this study, CycleGAN is selected as the foundational network model, upon which enhancements and reinforcement training are employed specifically for character images and landscape images. This approach is undertaken with the goal of bolstering the applicability of these techniques to forthcoming research endeavors in the domains of art education and dissemination.

CNN and ResNet

CNN and ResNet stand as pivotal neural network architectures within the domain of deep learning, primarily harnessed for computer vision tasks, including image classification, target detection, and image segmentation. CNNs, specifically, are purpose-built deep learning models adept at processing data with a grid-like structure, a prime example being images. They autonomously acquire salient image features through convolutional and pooling layers, subsequently leveraging fully connected layers to undertake tasks such as classification or regression. The core operation in the convolutional phase is encapsulated in Eq. (1): (1)${\displaystyle C\left(x,y\right)=\sum i\sum jI\left(x+i,y+j\right)K\left(i,j\right)}$

where C is the output after the convolution, and I(x,y) is the position of the image and K is the position of the kernel. Traditional CNN networks often suffer from gradient explosion and network is too deep and difficult to converge. For this reason, ResNet was proposed based on residual features (Wightman, Touvron & Jégou, 2021). ResNet’s fundamental concept revolves around facilitating the training of deep neural networks by enabling the direct transfer of information between different network layers through skip connections or jump connections. This structural design simplifies the network’s ability to effectively capture and learn a wide array of features, rendering it highly successful in image classification and a multitude of other computer vision tasks (Iqbal & Ali, 2018). The residual block, serving as the foundational building block, is visually depicted in Fig. 1:

The block consists of two weight layers and a jump join. Assuming the input is x, the output of the residual block can be expressed as: (2)${\displaystyle F\left(x\right)=H\left(x\right)+X}$

where F(x) is the output, x is the input, and H(x) is the residual function, which represents the residuals to be learned by the network. By adding x to H(x), the residual block enables the network to learn the residuals. In terms of gradient, by converting from cumulative multiplication to cumulative addition, the effectively overcomes the problem of gradient dispersion, which can be analyzed by Eq. (3): (3)${\displaystyle X_L=X_l+\sum _{i=1}^{L-1}F\left(x_i,w_i\right)}$

where X_L represents the gradient propagation for the next step and F is the output of the last output after the convolution.

Attention enhanced CycleGAN for the image style migration generation

The GAN embodies the concept of a zero-sum game, as rooted in game theory. In this GAN framework, two principal entities engage in a strategic game: the Generator (G) and the Discriminator (D) (Din et al., 2020). The primary objective of the Generator is to create synthetic data that closely matches the distribution of real sample data. Subsequently, the Generator submits both the generated data and genuine data to the Discriminator, which must discern whether the presented data originates from real or generated sources. In generative adversarial networks (GANs), generators create realistic synthetic data from random noise, aiming to deceive discriminators, which, in turn, strive to accurately distinguish between real and generated data. The adversarial interplay between these components leads to continuous improvement, culminating in the generation of high-quality, realistic data. In this process, the discriminator struggles to differentiate between real and synthetic samples, demonstrating the overall efficacy of GANs in data synthesis. The adversarial interplay between the Generator and the Discriminator in the GAN framework is formally captured in Eq. (4): (4)${\displaystyle \underset{\mathrm{G}}{min}\underset{\mathrm{D}}{max}\mathrm{V}\left(\mathrm{D},\mathrm{G}\right)={\mathrm{E}}_{\mathrm{x}\sim {\mathrm{P}}_{\text{data}}\left(\mathrm{x}\right)}\left[log\mathrm{D}\left(\mathrm{x}\right)\right]+{\mathrm{E}}_{\mathrm{z}\sim {\mathrm{P}}_{\mathrm{z}}\left(\mathrm{x}\right)}\left[log\left(1-\mathrm{D}\left(\mathrm{G}\left(\mathrm{x}\right)\right)\right)\right]}$

where x represents the real data, and z is noise, and G(z) is the generated data, and P_data(x) is the real data distribution, and P_z(x) denotes the noise distribution.

For the discriminator D, the cost function ${\mathrm{J}}^{\left(\mathrm{D}\right)}$ is of the form shown below: (5)${\displaystyle {\mathrm{J}}^{\left(\mathrm{D}\right)}\left({\theta }^{\left(\mathrm{D}\right)},{\theta }^{\left(\mathrm{G}\right)}\right)=-\frac{1}{2}{\mathrm{E}}_{\mathrm{x}\sim {\mathrm{P}}_{\text{data}}}log\mathrm{D}\left(\mathrm{x}\right)-\frac{1}{2}{\mathrm{E}}_{\mathrm{x}\sim {\mathrm{P}}_{\mathrm{z}}}log\left(1-\mathrm{D}\left(\mathrm{G}\left(\mathrm{z}\right)\right)\right)}$

A zero-sum game is played between the generator and the discriminator, the combined cost of the two is zero, and the cost functions of the generator and the discriminator are shown in Eq. (6): (6)${\displaystyle {\mathrm{J}}^{\left(\mathrm{G}\right)}=-{\mathrm{J}}^{\left(\mathrm{D}\right)}.}$

Thus it can be represented by a value function V to represent ${\mathrm{J}}^{\left(\mathrm{G}\right)}$ and ${\mathrm{J}}^{\left(\mathrm{D}\right)}$. (7)${\displaystyle \mathrm{V}\left({\theta }^{\left(\mathrm{D}\right)},{\theta }^{\left(\mathrm{G}\right)}\right)={\mathrm{E}}_{\mathrm{x}\sim {\mathrm{P}}_{\text{data}}}log\mathrm{D}\left(\mathrm{x}\right)+{\mathrm{E}}_{\mathrm{x}\sim {\mathrm{P}}_{\mathrm{z}}}log\left(1-\mathrm{D}\left(\mathrm{G}\left(\mathrm{z}\right)\right)\right)}$ (8)${\displaystyle {\mathrm{J}}^{\left(\mathrm{G}\right)}=-{\mathrm{J}}^{\left(\mathrm{D}\right)}=\frac{1}{2}\mathrm{V}\left({\theta }^{\left(\mathrm{G}\right)},{\theta }^{\left(\mathrm{D}\right)}\right)}$

By finding a suitable V-value function to make the cost function as small as possible, from completing the definition of the value function as a problem of solving very large and very small values. That is, the most discriminants D* and G for the problem shown in Eq. (8) are found: (9)${\displaystyle arg\underset{\mathrm{G}}{min}\underset{\mathrm{D}}{max}\mathrm{V}\left(\mathrm{D},\mathrm{G}\right)}$

CycleGAN, akin to GAN, operates as an unsupervised learning model, facilitating the migration of image styles across distinct domains. The model’s innovation lies in its ability to learn mappings between different domains without the need for paired training data. The integration of an attention mechanism further refines its performance by focusing on crucial regions during style translation, preserving intricate details. The inclusion of pre-training and comprehensive testing ensures robust generalization to diverse datasets and artistic styles. Through these features, CycleGAN accomplishes effective image style migration, making it a valuable tool for applications such as art education, where the unsupervised learning approach enables the model to autonomously acquire and replicate diverse artistic styles. The CycleGAN model, along with the DiscoGAN and DualGAN models, stands out for not requiring dataset matching to achieve cross-domain mapping relationships and the transmutation of different styles between domains. Within the CycleGAN architecture, two mapping functions are at play: G and F, complemented by two distinct discriminators, DX and DY. Here, X and Y denote two image datasets originating from different domains. The Generator G serves as the mapping from domain X to domain Y, while Generator F serves as the counterpart, mapping from domain Y to domain X (Gnanha et al., 2022).

CycleGAN uses a cyclic consistency loss such as in Eq. (10) L_cyc to represent this difference. (10)${\displaystyle \begin{array}{c}\hfill {\mathrm{L}}_{\mathrm{cyc}}\left({\mathrm{G}}_{\mathrm{c}\to \mathrm{p}},{\mathrm{G}}_{\mathrm{p}\to \mathrm{c}}\right)={\mathrm{E}}_{\mathrm{c}{\mathrm{p}}_{\text{data}}\left(\mathrm{c}\right)}\left[{∥{\mathrm{G}}_{\mathrm{p}\to \mathrm{c}}\left({\mathrm{G}}_{\mathrm{c}\to \mathrm{p}}\left(\mathrm{c}\right)\right)-\mathrm{c}∥}_1\right]\hfill \\ \hfill +{\mathrm{E}}_{\mathrm{p}{\mathrm{p}}_{\text{data}}\left(\mathrm{p}\right)}\left[{∥{\mathrm{G}}_{\mathrm{c}\to \mathrm{p}}\left({\mathrm{G}}_{\mathrm{p}\to \mathrm{c}}\left(\mathrm{p}\right)\right)-\mathrm{p}∥}_1\right]\hfill \end{array}}$

where, the G_c→p denotes the forward generator i.e., G, and G_p→c denotes the backward generator i.e., F. The cyclic process of CycleGAN is shown in Fig. 2:

The generator structure under this network structure is shown in Fig. 3:

The generative network initiates its operation by downsampling the image through an encoder before entering the intermediate transformation model. Within the intermediate transformation model, the image undergoes four encoding residual blocks to derive the feature map. Subsequently, the feature map is channeled into the attention module, where the attention mechanism is introduced at the encoding stage. The integration process of the attention module into the encoding stage is visually depicted in Fig. 4:

As illustrated in Fig. 4, this attention-weighting module introduces attention through channel weighting. Initially, the encoded feature map undergoes global average pooling (GAP) and global maximum pooling (GMP) procedures, enabling the extraction of both global and fine-grained texture information from the image, respectively. Global average pooling is achieved through sliding window averaging across the feature map, while global maximum pooling involves selecting the maximum value for each feature map Finally, the feature map is multiplied by learned parameter weights, assigning distinct weights to each channel within the encoder feature map. The magnitude of these weights determines the channel’s significance in the feature, effectively introducing the attention mechanism (Yan et al., 2022).

Building upon this foundation, we establish an evaluation framework for the image style migration model based on the proposed ATT-CycleGAN. The overall framework is outlined in Fig. 5.

Style migration through ATT-CycleGAN is realized using pertinent public datasets and the art teaching images employed in this study.

Datasets

For the experimental data, two distinct publicly available datasets were chosen to facilitate comparative testing in this study. Given the specific focus on style migration within the domain of art, the primary emphasis is placed on style transformation tasks and the migration of landscape object styles. To that end, the selfie2anime dataset (Torbunov et al., 2023) is utilized for data analysis in the style transformation process, while the image styles are acquired from the Places365 dataset (Zhou et al., 2017) for both training and testing purposes. The Places365 dataset is a comprehensive scene recognition dataset encompassing over 1.8 million images across 365 scene categories (https://zenodo.org/records/5926442). In this article, we select a subset of 1,000 paintings from the dataset, as provided by Svoboda, for model testing. The selfie2anime dataset, on the other hand, comprises selfies and comics (https://zenodo.org/records/10130356). For the selfie component, over 40,000 selfies are available, with only women’s photos utilized for training. The anime section involves over 20,000 face images, and for this dataset, female anime faces from the manga section are chosen. Similarly, for this data selection, 1,000 samples are selected for training and testing. Additionally, 100 images are specifically curated, segregating them into sketch style and original images to align with the design requirements of the art teaching and communication model. The information for these datasets can be summarized as shown in Table 1. All dataset splits are executed at a 7:3 ratio.

To assess the model’s performance, classic bilinear interpolation (Gribbon & Bailey, 2004), the approach proposed by Johnson, Alahi & Fei-Fei (2016), the method proposed by Han et al. (2018), as well as the original unprocessed CycleGAN, are selected for method comparisons in model validation.

Experiment setup and details

Structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) metrics are used to evaluate the model performance during the selection process of evaluation metrics (Hore & Ziou, 2010), and the above performances are calculated as shown in Eqs. (11) and (12): (11)${\displaystyle \text{SSIM}\left(\mathrm{X},\mathrm{Y}\right)=\frac{\left(2{\mu }_{\mathrm{X}}{\mu }_{\mathrm{Y}}+{\mathrm{C}}_1\right)\left(2{\sigma }_{\mathrm{XY}}+{\mathrm{C}}_2\right)}{\left({\mu }_{\mathrm{X}}^2+{\mu }_{\mathrm{Y}}^2+{\mathrm{C}}_1\right)\left({\sigma }_{\mathrm{X}}^2+{\sigma }_{\mathrm{Y}}^2+{\mathrm{C}}_2\right)}}$

where X and Y denote the images to be measured, the μ_X and μ_Y denote the images X and image Y are the mean values of the ${\sigma }_{\mathrm{X}}^2$ and ${\sigma }_{\mathrm{Y}}^2$ represent the mean values of the images X and the covariance of Y the covariance of C₁ and C₂ The value of SSIM ranges from 0 to 1. The closer the value is to 1, the higher the similarity between the two images, and the more serious the image distortion is. PSNR is an important index for evaluating the quality of the image, and its calculation is shown in Eq. (12). (12)${\displaystyle \begin{array}{c}\hfill \mathrm{MSE}=\frac{1}{\mathrm{H}\times \mathrm{W}}\sum _{\mathrm{i}=1}^{\mathrm{H}}\sum _{\mathrm{j}=1}^{\mathrm{W}}{\left(\mathrm{X}\left(\mathrm{i},\mathrm{j}\right)-\mathrm{Y}\left(\mathrm{i},\mathrm{j}\right)\right)}^2\\ \hfill \text{PSNR}=10\times log10\left(\frac{{\left(2^{\mathrm{n}}-1\right)}^2}{\mathrm{MSE}}\right)\end{array}}$

where, the MSE denotes the image X and image Y The mean square error between H and W denote the height and width of the image; and n denotes the number of bits per pixel, and n The larger the value of PSNR, the smaller the distortion of the image, which means that the image has higher quality.

After completing the selection of data sets for the model and the establishment of relevant evaluation indicators, the modeling algorithm of our proposed method is as follows:

Experiment results and analysis

After completing the training process, as outlined in Algorithm 1 , and obtaining the respective models based on the selected dataset and the self-constructed dataset, we conducted model testing and analysis. In the context of model comparison, a comprehensive evaluation was carried out, encompassing the unimproved CycleGAN model, along with the methodologies discussed in Section ‘Datasets’. The results obtained from the selfie2anime dataset are shown in Table 2 and Fig. 6.

Based on the SSIM and PSNR metrics employed in the article, it is evident that the model proposed in this study achieves a more balanced and improved performance after the integration of the attention mechanism. The results across the two metrics demonstrate a superior and more consistent performance. After conducting the analysis on the selfie2anime dataset, a similar analysis was extended to the Places365 dataset, with the obtained results presented in Table 3 and Fig. 7.

The comparative results presented above highlight the superior performance of the method proposed, which extends to the Places365 dataset as well. The higher SSIM scores under the Places365 dataset compared to the selfie2anime dataset suggest that the proposed method excels in handling landscape-class images. This underscores its robust image analysis capabilities, particularly when applied to more complex visual content. After concluding the analysis of the two public datasets, an additional analysis was conducted on the self-constructed dataset, with the results depicted in Table 4 and Fig. 8.

The self-constructed datasets featured exclusively consist of landscape images. The outcomes from these datasets closely align with those of the Places365 dataset, signifying the inherent advantages of CycleGAN-like models for handling data with higher pixel counts and richer visual content. Furthermore, the proposed method maintains its superiority over other methods even when an attention mechanism is integrated. Additionally, this study conducted data testing with varying batch sizes, and the results are visually presented in Fig. 9,

Figure 9 primarily serves to contrast the performance of the proposed method with the CycleGAN method, which lacks the attention mechanism across different datasets and varying inputs. The boxplot visualizations illustrate that, following the integration of the attention mechanism, the proposed method consistently outperforms the counterpart model in terms of both SSIM and PSNR metrics. These findings affirm that the proposed method excels in both performance and stability, showcasing the advantages of the attention mechanism’s incorporation.