HCGAN: hierarchical contrast generative adversarial network for unpaired sketch face synthesis

Traditional methods

Traditional sketch synthesis approaches aim to learn the mapping relationship between images and sketches. This method mainly includes Bayesian inference, representation learning, and subspace learning models.

Bayesian inference models update the sketch localization based on a probabilistic model using real data. For example, Chen et al. (2001) proposed an instance-based sketch face synthesis system that utilizes a nonparametric sampling algorithm to learn sketching style details. Nefian & Hayes (1999) used an embedded Hidden Markov model to capture the nonlinear relationship between picture and sketch pairs. Wang et al. (2013) introduced a transduction learning method for synthetic sketching, employing a dynamic process to minimize the loss of a given test sample. Furthermore, Peng et al. (2015) devised a superpixel approach based on Markov models to enhance flexibility without dividing photos into regular rectangular blocks. However, these traditional methods often produce suboptimal results, partly because they rely on manually crafted features and do not always capture the complexity of the mapping relationship.

Subspace learning models aim to transform high-dimensional spatial features into low-dimensional ones. Tang & Wang (2002) proposed a series of example-based methods based on linear feature transformation techniques. However, these methods rely on global linear systems and fail to capture the complex relationship between photo and sketch pairs fully. To address this issue, Liu et al. (2005) used the local linear embedding (LLE) method, which ensures locally geometrically similar stream shapes for photo and sketch patches in two distinct image spaces. Nevertheless, this method separates pseudo-sketch generation and representation learning into two distinct processes, resulting in suboptimal outcomes.

Huang & Wang (2013) introduced a joint learning framework encompassing domain-specific dictionary learning and subspace learning. The representation learning model primarily relies on sparse coding with dictionary learning. Ji et al. (2011) highlighted the limitations of capturing personalized features through a synthetic process. Wang et al. (2012) proposed a semi-coupled dictionary learning method, employing a linear transformation to bridge the gap between two different domain-specific representations. Gao et al. (2012), building on a two-step algorithm. Zhang et al. (2011), presented a selection scheme for generating initial pseudo-images and introduced sparse representation-based enhancement (SRE) for sketch synthesis.

However, these methods involve extensive calculations, resulting in poor real-time performance and limited practicality. Moreover, they rely on large-scale training data, making achieving satisfactory results with limited training data challenging.

Deep learning-based methods

Deep learning-based methods have become increasingly popular for sketch synthesis due to their ability to train on datasets and generate high-quality results. Among these methods, generative adversarial networks (GANs) are widely used.

Zhang et al. (2015) introduced the first deep photo-sketch synthesis model using a fully convolutional neural network (FCNN), but struggled to preserve certain details. To mitigate this issue, Zhang et al. (2018a) proposed PGAN, which uses a specific parametric Sigmoid activation function to reduce the impact of photo prior and lighting variations. Similarly, Wang & Sindagi (2018) introduced a synthesis method called PS2MAN, which employs two U-Net architectures within a multiple adversarial network to generate high-quality images at varying resolutions. To address blurring and distortion, Zhang et al. (2018b) proposed a multi-domain adversarial learning (MDAL) approach to sketch face synthesis. Liang et al. (2023) proposed Parallel Multistage GANs for Face Image Translation (PMSGAN), achieving synthesis from coarse to fine.

More and more methods are introduced of identity-aware models (Fang et al., 2020; Lin et al., 2020), incorporating novel perceptual losses to train image generation models with facial recognition as the ultimate goal. Other notable methods include Yu et al. (2020), who introduced a synthesis-assisted generative adversarial network utilizing facial synthesis information, and Duan et al. (2020), who implemented a multiscale self-attentive residual learning framework for face photo-sketch conversion. Another notable approach (Goodfellow et al., 2020) does not require any source domain images for training and utilizes deep features extracted from CNNs and manual features.

Moreover, researchers increasingly emphasize the importance of preserving the content features of the input optical image while modifying stylistic features in sketch-based face synthesis. Seo, Ashtari & Noh (2023) introduced a style encoder based on Cycle-GAN to refine the network’s control over styles. Cui et al. (2021) proposed the Self-Supervised Semantic Network (SSNet), incorporating both style and semantic feature encoding. Gao et al. (2021) introduced CHAN, reinforcing the network’s perceptual capabilities for features such as textures. Li et al. (2022) introduced color refinement loss and texture loss, further augmenting the network’s control over content features. Kong et al. (2023) proposed an Asymmetric Double-Stream Generative Adversarial Network (ADS-GAN), incorporating edge regularization constraints. Yun et al. (2024) proposed a novel sketch face synthesis method called StyleSketch, based on prior knowledge. It utilizes pre-trained StyleGAN to extract rich semantic deep features, achieving satisfactory results.

It is worth mentioning that diffusion (Sohl-Dickstein et al., 2015), a newer and better general style transfer model, is currently available. Diffusion models is to find a noise map and a conditioning vector corresponding to a generated image. It is a potential way to improve the quality of example-guided artistic image generation. Dhariwal & Nichol (2021) invert the deterministic DDIM (Song, Meng & Ermon, 2020) sampling process in closed form to obtain a latent noise map that will produce a given real image. Ramesh et al. (2022) develop a text-conditional image generator based on the diffusion models and the inverted CLIP. The above methods are difficult to generate new instances of a given example while maintaining fidelity. Zhang et al. (2023) propose an inversion-based style transfer method (InST), which can efficiently and accurately learn the key information of an image, thus capturing and transferring the artistic style of a painting. However, the above diffusion-based methods cannot convert optical images into sketch images when the training set size is small. The reason is that when the training data is small, it is difficult for them to correct inappropriate weights in the pre-training weights, making it difficult to ensure that the style features of the optical image are completely removed, resulting in the failure of the sketch face synthesis task.

Detailed sketches can only be drawn manually by painters, resulting in sketched face datasets that are often small in size. When training data is small, GAN-based methods are often the most effective. In addition, compared with the Diffusion-based method, the GAN-based method has the advantages of small parameter size and fast optimization. However, although the above GAN-based methods have synthesized high-quality sketches, local details are still unsatisfactory, and the dependence on paired training data limits the broadening of application scenarios of most methods.

Overview

Our article introduces a two-module generative adversarial network framework, as depicted in Fig. 1, incorporating novel loss functions for training a network F(⋅) to produce high-quality sketches from input images. The framework comprises a global sketch synthesis module and a local sketch refinement module, each equipped with dedicated generators, discriminators, and block feature extractors.

The global module generates overall sketches, while the local module focuses on extracting accurate local features. However, solely relying on adversarial loss neglects image details, which becomes evident in unpaired training data scenarios, resulting in pseudo-sketch faces with insufficient local details. To address this limitation, we propose multi-layer block contrast loss, local consistency loss, and local refinement loss to generate refined local details in the sketch. This enhancement leads to pseudo-sketches with more precise details. Moreover, the framework can be trained end-to-end without requiring post-processing.

In our article, we refer to the four local generators, discriminators, and block feature extractors as G_i, D_i, and H_i, respectively.

Global sketch synthesis module

The global sketch synthesis module mainly consists global generator G, global discriminator D, and global block feature extractor H. The block feature extractor H and the encoding part E of the global generator G form the global multi-layer block comparison module. The global sketch synthesis module takes the unpaired image pair $\left\{P_i,S_j\right\}$ as input. The optical face image P_i is passed through the global generator G to obtain pseudo-sketch $S_i^{\prime }=G\left(P_i\right)$. Meanwhile the true sketch S_j and the pseudo sketch $S_i^{\prime }$ are input to the global discriminator D to assess the authenticity of the pseudo sketch $S_i^{\prime }$.

The generator G used in this article is based on the generator in Johnson, Alahi & Fei-Fei (2016) and includes residual blocks in its middle layer. This mitigates the problem of network degradation that occurs when increasing the number of network layers. Incorporating residual blocks reduces the loss of crucial features and improves the effectiveness of network training, resulting in the generation of more realistic and high-quality sketches.

The discriminator D used in this article is based on the discriminator in Isola et al. (2017). This discriminator focuses solely on the local structure of the image, allowing it to effectively learn the high-frequency information and detailed features of the image. This approach reduces the parameters required during training and improves training efficiency.

The block feature extractor H in HCGAN is based on the architecture of the block feature extractor used in SimCLR (Chen et al., 2020), which includes two fully connected layers. After the encoder extracts features from the optical or sketch image, the block feature extractor H stacks these features in blocks, creating conditions for subsequent multi-layer block contrast loss.

The global sketch synthesis module utilizes global adversarial loss and global multi-layer block contrast loss for optimization. Additionally, in the latter stage of training, it introduces local refinement loss (as mentioned in ‘Local refinement loss’) to enhance the optimization process further.

Global adversarial loss L_GAN aims to synthesize sketches with the sketch style under unpaired input. It establishes a “gaming” optimization process between the global generator G and the global discriminator D, enabling the global generator G to fully learn the input sketch style and enhance the composite image’s quality. The equation of L_GAN is as follows. (1)${\displaystyle L_{GAN}\left(G,D,P,S\right)=E_{P_i\sim P}log\left(1-D\left(G\left(P_i\right)\right)\right)+E_{S_j\sim S}logD\left(S_j\right)}$

Global multi-layer block contrast loss L_MBCG is inspired by the literature Park et al. (2020) to maximize the mutual information between the output sketch and the input image. It could be helpful to improve the composite image’s quality. This loss is similar to the local multi-layer block contrast loss L_MBCL in Eq. (6) and multi-layer refinement loss L_MBCP in Eq. (11). They are all based on multi-layer block contrast loss.

The loss of multi-layer block comparison is presented in Fig. 2, which corresponds to both the global and the local multi-layer block comparisons as shown in Fig. 1. We obtain an image block (or a query block) from the encoding feature of each layer of the pseudo-sketch. From the input optical image, we select an image block (or a positive class block) with the same position as the coding feature of each layer. The rest of the image blocks, which have different positions from the coding features of each layer, are considered negative class blocks. The query block is expected to have a strong association with the positive class block and a minor association with the negative ones. To construct N+1 classification, we crop N negative class blocks at different positions of the input optical image paired with a positive class block. We employ the Cross-Entropy loss to maximize the association between the query block and the positive class block. The loss is formulated as follows. (2)${\displaystyle l\left(v,v^{+},v^{-}\right)=-log\left[\frac{exp\left(v•v^{+}/\tau \right)}{exp\left(v•v^{+}/\tau \right)+\sum _{n=1}^{N}exp\left(v•v^{-}/\tau \right)}\right]}$ where, v denotes the query block, v⁺ denotes the positive class block, v⁻ denotes the negative class block, and τ denotes the scaling factor mentioned in Park et al. (2020). Specifically, we set τ to 0.07 in the experiments conducted in this article.

The global multi-layer block contrast module provides the global multi-layer block contrast loss. To compute this loss, we feed the input optical image P_i and the output pseudo-sketch image $S_i^{\prime }$ into the encoder E. We then use the output of the middle layer of the encoder as the input to the block feature extractor H. The block feature extractor H chunks the input, creating a classification problem for each layer, and calculates the Cross-Entropy loss to obtain the global multi-layer block contrast loss, denoted by L_MBCG. (3)${\displaystyle L_{MBCG}\left(G,H,P\right)=E_{P_i\sim P}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{x}}_m^n,x_m^n,x_m^{N-}\right)}$ where, (4)${\displaystyle {\hat{x}}_m^n={\left\{H\left(E_m\left(S_i^{\prime }\right)\right)\right\}}_n={\left\{H\left(E_m\left(G\left(P_i\right)\right)\right)\right\}}_n}$ where, $E_m\left(S_i^{\prime }\right)$ represents the output of m-th layer of the pseudo-sketch obtained via the encoder E. Similarly, ${\hat{x}}_m^n$ denotes the n-th block, which is obtained by chunking $E_m\left(S_i^{\prime }\right)$ using the block feature extractor H. On the other hand, $x_m^n={\left\{H\left(E_m\left(P_i\right)\right)\right\}}_n$ implies that the m-th layer output of the input optical image is passed through the encoder E, and then chunked via the block feature extractor H to yield the positive class block (with sequence number n). $x_m^{N-}={\left\{H\left(E_m\left(P_i\right)\right)\right\}}_{!n}$ denotes the blocks obtained after dividing the output of the m-th layer of encoder E of the input optical image via the block feature extraction network H. These blocks exclude the positive class blocks (negative class blocks), as identified by the symbol “!” in “!n”. The symbol “!n” indicates the negation of “n”, i.e., all blocks except the one with the sequence number n. The Cross-Entropy loss operation described in Eq. (2) is represented by l.

Local sketch refinement module

The local sketch refinement module comprises four generators G_i, four discriminators D_i, and four local block feature extractors H_i. It takes in four local regions extracted from the optical image P_i, the sketch image S_j, and the pseudo-sketch $S_i^{\prime }$. The main objective of this module is to provide adversarial refinement loss L_GANP, multi-layer refinement loss L_MBCP, and local consistency loss L₁. These three losses help the global generator G produce more locally refined pseudo-sketches $S_i^{\prime }$.

The local multi-layer block comparison module uses the encoding part E_i of the local generator G_i as its encoder. To obtain more targeted features for different facial parts, we trained four local generators, denoted by G_i. Specifically, each local generator G_i accepts one of four parts of the optical image P_i: left eye P_iel, right eye P_ier, nose P_in, and mouth P_im. It then generates the respective local pseudo-sketches $S_{iel}^{\prime }$, $S_{ier}^{\prime }$, $S_{in}^{\prime }$ and $S_{im}^{\prime }$ for each region.

To improve the performance of the local multi-layer block comparison module and the local discriminator, we optimize them using two loss functions: the local adversarial loss and the local multi-layer block contrast loss. These loss functions aim to generate more precise and visually appealing local pseudo-sketches with the local generator G_i. Conversely, the role of the local discriminator D_i is to differentiate between real and generated local sketches. By optimizing these components, we aim to enhance the module’s ability to extract local features.

Local adversarial loss L_GANL is based on adversarial loss. We utilize a competitive optimization approach between the local generator and local discriminator using local adversarial loss to enhance the local feature encoding of the encoding part of the local generator. This method helps us to obtain optimized local generator encoding parts and local discriminators, which are essential for the local refinement loss. (5)${\displaystyle L_{GANL}\left(G_i,D_i,P,S\right)=E_{P_{iel}\sim P}log\left(1-D_{iel}\left(G\left(P_{iel}\right)\right)\right)+E_{S_{jel}\sim S}log{D}_{iel}\left(S_{jel}\right)+E_{P_{ier}\sim P}log\left(1-D_{ier}\left(G\left(P_{ier}\right)\right)\right)+E_{S_{jer}\sim S}log{D}_{ier}\left(S_{jer}\right)+E_{P_{in}\sim P}log\left(1-D_{in}\left(G\left(P_{in}\right)\right)\right)+E_{S_{jn}\sim S}log{D}_{in}\left(S_{jn}\right)+E_{P_{im}\sim P}log\left(1-D_{im}\left(G\left(P_{im}\right)\right)\right)+E_{S_{jm}\sim S}log{D}_{im}\left(S_{jm}\right)}$

Local multi-layer block contrast loss L_MBCL is similar to L_MBCG in Eq. (3). It is particularly useful for improving the performance of the local generator by enhancing the correlation between the input optical local feature and the output sketch local feature. This improvement enables a better encoding part for the local generator, facilitating enhanced capture and encoding of local features. (6)${\displaystyle L_{MBCL}\left(G_i,H_i,P\right)=E_{P_{in}\sim P}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{y}}_{min}^n,y_{min}^n,y_{min}^{N-}\right)+E_{P_{im}\sim P}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{y}}_{mim}^n,y_{mim}^n,y_{mim}^{N-}\right)+E_{P_{ier}\sim P}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{y}}_{mier}^n,y_{mier}^n,y_{mier}^{N-}\right)+E_{P_{iel}\sim P}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{y}}_{miel}^n,y_{mie}^n,y_{mie}^{N-}\right).}$

As with the global multi-layer block contrast loss L_MBCG, we obtain ${y_m^n}_{in}$ for the nose part P_in by consecutively feeding the input through the local generator G_in and the nose local block feature extractor H_in. The equation is as follows. (7)${\displaystyle {\hat{y}}_{min}^n={\left\{H_{in}\left(E_{inm}\left(S_{in}^{\prime }\right)\right)\right\}}_n={\left\{H_{in}\left(E_{inm}\left(G_{in}\left(P_{in}\right)\right)\right)\right\}}_n}$ Moreover: (8)${\displaystyle y_{min}^n={\left\{H_{in}\left(E_{inm}\left(P_{in}\right)\right)\right\}}_n{y}_{min}^{N-}={\left\{H_{in}\left(E_{inm}\left(P_{in}\right)\right)\right\}}_{!n}}$ Similarly, the respective characters in multi-layer refinement loss L_MBCP in Eq. (11) can be obtained. (9)${\displaystyle {\hat{z}}_{min}^n={\left\{H_{in}\left(E_{inm}\left(S_{iin}^{\prime }\right)\right)\right\}}_n{z}_{min}^n={\left\{H_{in}\left(E_{inm}\left(P_{in}\right)\right)\right\}}_n{z}_{min}^{N-}={\left\{H_{in}\left(E_{inm}\left(P_{in}\right)\right)\right\}}_{!n}}$

Local refinement loss operates on the global sketch synthesis module and is constrained by both multi-layer features and realism, ensuring the synthesized sketch exhibits more accurate local feature expression. It includes multi-layer refinement loss L_MBCP and adversarial refinement loss L_GANP as follows.

Adversarial refinement loss L_GANP is optimized between the global generator and the local discriminators. We optimize the details of sketches generated by the global generator using the global sketch synthesis module. (10)${\displaystyle L_{GANP}\left(G_i,D_i,P,S\right)=E_{P_{iel}\sim P}log\left(1-D_{iel}\left(S_{iiel}^{\prime }\right)\right)+E_{S_{iel}\sim S}log{D}_{iel}\left(S_{jel}\right)+E_{P_{ier}\sim P}log\left(1-D_{ier}\left(S_{iier}^{\prime }\right)\right)+E_{S_{ier}\sim S}log{D}_{ier}\left(S_{jer}\right)+E_{P_{in}\sim P}log\left(1-D_{in}\left(S_{iin}^{\prime }\right)\right)+E_{S_{in}\sim S}log{D}_{in}\left(S_{jn}\right)+E_{P_{im}\sim P}log\left(1-D_{im}\left(S_{iim}^{\prime }\right)\right)+E_{S_{im}\sim S}log{D}_{im}\left(S_{jm}\right)}$

Multi-layer refinement loss L_MBCP is introduced to constrain the association between the parts of the generated sketches and the corresponding input. This loss is necessary to prevent the loss of crucial local information in the generated sketches. (11)${\displaystyle L_{MBCP}\left(G_i,H_i,P\right)=E_{P_{in}\sim P,S_{iin}^{\prime }\sim S_i^{\prime }}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{z}}_{min}^n,z_{min}^n,z_{min}^{N-}\right)+E_{P_{im}\sim P,S_{iim}^{\prime }\sim S_i^{\prime }}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{z}}_{mim}^n,z_{mim}^n,z_{mim}^{N-}\right)+E_{P_{ier}\sim P,S_{iier}^{\prime }\sim S_i^{\prime }}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{z}}_{mier}^n,z_{mier}^n,z_{mier}^{N-}\right)+E_{P_{iel}\sim P,S_{iiel}^{\prime }\sim S_i^{\prime }}\sum _{m=1}^M\sum _{n=1}^{N+1}l\left({\hat{z}}_{miel}^n,z_{miel}^n,z_{mierl}^{N-}\right)}$

Association strategy

Optimization balancing is a recurring challenge in multi-module networks, including HCGAN, which consists of a global sketch synthesis module and a local sketch refinement module. Additionally, HCGAN involves local refinement loss between two modules, which poses a challenge for optimization balancing. To deal with this challenge, this article proposes two methods: local consistency loss and “warmup-epoch” strategy.

Local consistency loss L₁ ensures that the correlation between the two modules is optimized and addresses the issue of significant differences between the locally generated sketches by the global generator G and the local generator module G_i. This discrepancy reduces the effectiveness of L_MBCP and L_GANP in optimizing the global generator G. (12)${\displaystyle L_1=∥S_{iel}^{\prime }-S_{iiel}^{\prime }∥+∥S_{ier}^{\prime }-S_{iier}^{\prime }∥+∥S_{in}^{\prime }-S_{iin}^{\prime }∥+∥S_{im}^{\prime }-S_{iim}^{\prime }∥}$

The “warmup-epoch” strategy optimizes the relation between local feature extraction and global optimization and is inspired by an approach described in the literature (Wu et al., 2020; Lyu, Rosin & Lai, 2023). The proposed model utilizes a global sketch synthesis module to generate pseudo-sketches. Additionally, the local sketch refinement module provides local refinement loss to enhance local details. By implementing the “warmup-epoch” strategy, the local sketch refinement and global sketch synthesis modules are initially optimized separately. This allows the global sketch synthesis module to generate higher-quality global sketches, enabling the local sketch refinement module to distill local features better. When an improved global sketch is generated, the multi-layer refinement loss and adversarial refinement loss provided by the local sketch refinement component are employed to optimize the global generator G for local detailing.

The total loss function of the network is: (13)${\displaystyle L_{total}={a\times L}_{GANL}+{b\times L}_{MBCL}+{c\times L}_{GANP}+{d\times L}_{MBCP}+e\times L_{GAN}+{f\times L}_{MBCG}+L_1.}$

According to the “warmup-epoch”, we divide the training process into two stages. In the first stage of this experiment, c = d = 0, a = b = 0.25, e = f = 1. In the second stage of the training, a = b = c = d = 0.25, e = f = 1. It is worth mentioning that, given that sketches, as facial images, require particular attention to the coordination among different components at the global-level, a higher weight is allocated to the loss associated with global sketching in the overall loss function. In comparison, a lower weight is assigned to the loss linked to local refinement to avoid excessive local refinement leading to overall imbalance.

Evaluation metrics

Currently, many metrics are used for image quality assessment. To better simulate human visual perception, we choose to use Fréchet Inception Distance (FID) (Heusel et al., 2017) along with Content Fidelity (CF), Global Effects (GE), and Local Patterns (LP) (Wang et al., 2021) to evaluate the test results.

FID is a metric to assess the quality of images generated by generative models. It is calculated by comparing the feature distributions of generated images with those of real images. Specifically, FID utilizes a deep neural network (typically a pre-trained Inception network) to extract features from images and then computes the Fréchet distance between the feature distributions of generated and real images. This distance measures the similarity between the two feature distributions.

The combination of CF, GE, and LP can effectively and comprehensively evaluate the quality of style transfer generated images. Leveraging feature extraction networks, they simulate human perception of images by focusing on content, global, and local aspects, respectively. GE emphasizes global aspects such as global colors(GC) and holistic textures(HT). Similarly, LP consists of two parts: one is to assess the similarity of local pattern counterparts directly, and the other is to compare the diversity of retrieved pattern categories.

Overall, by combining the evaluation metrics of FID, CF, GE and LP. We can better simulate human visual perception and comprehensively evaluate the quality of generated images, thus providing important guidance and reference for improving image generation technology.

Dataset and setting

Since high-quality sketches can only be produced by hand, obtaining them is expensive and time-consuming. Therefore, current high-quality sketch datasets tend to be smaller in size. In order to fully demonstrate the effectiveness and robustness of HCGAN, the experiment section employs the CUFS and SKSF-A datasets.

The CUFS datasets comprise the CUHK, AR, and XM2VTS datasets. CUHK and AR datasets have fewer accessories, and both of them are frontal photos. There are many accessories, such as glasses and earrings, in the XM2VTS dataset. Each dataset contains a varying number of image-sketch pairs, with 188, 123, and 295 pairs, respectively. Image-sketch pairs consist of an optical photograph paired with a corresponding sketch hand-drawn by painters. The SKSF-A dataset provides 134 optical images along with corresponding sketches in seven different styles. And there is a large difference in angle. However, most of these styles are simple sketches or initial drafts, which are not suitable for training a fine-sketch synthesis model. Therefore, we only selected the style with the highest sketch quality for our experiments. The training and test sets include different numbers of optical image-sketch pairs for each dataset. The division of training and testing sets for each dataset is shown in Table 1.

Before the experiment, it is necessary to process the data. The face images and corresponding sketches from the CUFS dataset were aligned by the Multi-task Cascaded Convolutional Neural Networks (MTCNN). Furthermore, the optical images in the SKSF-A dataset featured diverse backgrounds, which could negatively impact the synthesis outcomes. Thus, we employed masks provided by the SKSF-A dataset to crop the images, replacing the backgrounds with white.

We conducted training and testing on a server equipped with an Intel(R) Xeon(R) CPU E5-2640 v4 and an NVIDIA GeForce RTX 4090. During training, a minimum of 6GB of GPU memory was required. The network is trained with a batch size of 1. The training optimizer is the Adam optimizer, and the momentum parameters β₁ and β₂ are set to 0.9 and 0.999, respectively. The training period consists of 800 epochs, with the learning rate set to 0.0002 for the first 400 epochs and gradually decaying from 0.0002 to 0 for the following 400 epochs. In order to avoid the slow model training caused by too many operations, only the outputs of 0, 4, 8, 12 and 16 layers of the encoder are used in the calculation of L_MBCG, L_MBCL, and L_MBCP, and the number of blocks in each layer is 64.

Ablation experiments

To verify the effectiveness of the network proposed in this article, this section removes different components from HCGAN, including the local sketch refinement module, the local consistency loss L₁, adversarial refinement loss L_GANP, multi-layer refinement loss L_MBCP and “warmup-epoch” to examine their impact on the final synthesis results. Figure 3 illustrates the experimental results of method validity for different datasets, while Tables 2 and 3 compares the quantitative validity indices across different datasets.

The results are presented in Table 2 indicates that the complete model outperforms individual ablation experiments. Specifically, after adding the local sketch refinement module, the FID metric is reduced by about 26 under the CUHK dataset, 24 under the AR dataset, 60 under the XM2VTS dataset, and 25 under the SKSF-A dataset. Table 3 shows that the complete model performs best in LP, while maintaining good CF and GE. Additionally, the synthetic image quality of each ablation experiment was lower than that of the overall model.

Removing the local sketch refinement module (LSRM) leads to the global generator lacking optimization constraints related to local refinement loss. Consequently, this results in poor synthesis of local details in the pseudo-sketches, as evidenced by significantly lower LP scores. For instance, as depicted by the images in Fig. 3C, the eyes and mouth areas suffer from severe blurring. Additionally, the removal of L₁ results in a lack of correlation between the local and global generators, reflected by fluctuations in the CE, GE, and LP metrics. There is a decrease in the quality of synthesized local features, as shown in the eye portion of the image with shadows in Fig. 3D. Removing L_GANP and removing L_MBCP result in insufficient local constraints on the global generator. It makes the generated pseudo-sketches poorly represented in the eyes, mouth, and nose parts, as reflected by significantly lower LP scores. The results of removing the “warmup-epoch” are shown in the four images corresponding to Fig. 3G. The absence of the “warmup-epoch” leads to a mismatch between local and global optimization, as evidenced by the inability to obtain better global results in the early stages. For example, the testing results under the AR dataset are severely missing at the end of the hair. Better local results could not be obtained later, such as the testing results under the CUHK and SKSF-A datasets with more severe shadows between the two lips. Additionally, the robustness of the network decreases, as evidenced by significant fluctuations in the FID, CF, GE, and LP metrics after each training session.

Based on the results presented in Fig. 3, Tables 2 and 3, it is evident that each component of the experimental design contributes to the enhanced sketch synthesis. The complete model yields a higher-quality synthesis effect compared to each individual ablation experiment.

Comparison experiments

To verify the effectiveness and robustness of the HCGAN, this section compares it with existing methods, such as Cycle-GAN (Zhu et al., 2017) and pix2pix (Lin, Pang & Xia, 2020), among others. While FSGAN (Fan et al., 2022) adopts a similar global and local approach, Cycle-GAN and DRIT++ (Lee et al., 2018) use unpaired input, whereas pix2pix and FSGAN use paired input. Specifically, we also compare with the latest sketch face synthesis method, StyleSketch (Yun et al., 2024). The results for each dataset are presented in Figs. 4, 5, 6 and 7, and the quantitative metrics for each method on each dataset are shown in Tables 4 and 5.

Figures 4, 5, 6 and 7 show the comparison of the synthesis effect of each method under the three datasets, from left to right, for the optical image, the corresponding real sketch image, Cycle-GAN, pix2pix, DRIT++, FSGAN, HCGAN and StyleSketch respectively. As the figures show, Cycle-GAN relies on reconstruction to optimize the style transfer under unpaired input and obtains better global information. However, too many unnecessary pixel details are generated, and too much noise interferes, resulting in a less clear overall sketch image and poor visual perception. When processing the AR dataset, the eyes were severely deformed in the synthetic results. The pix2pix method relies on conditional GANs for sketch synthesis, aided by L₁ loss optimization. However, the synthesized sketches are severely distorted and the results are unsatisfactory. The main reason is that optical and real sketches are not strictly pixel-aligned, and using L₁ loss leads to shifting and blurring key details. DRIT++ is a general method for style transfer, but it suffers from the problem of contour ghosting in composite sketches. In the XM2VTS dataset, partial distortion exists in the synthesized sketches’ eyes despite the overall improvement in results. FSGAN splits and migrates the local part before fusion, creating a clear local portrayal. However, when processing XM2VTS, synthetic images have heavy shadows at the junction of local and other parts, and the double chin details are lost. Additionally, clothing edges lack global aspects, and the FID of synthetic sketches is poor. StyleSketch performs well on the CUHK and SKSF-A datasets but performs poorly on the AR and XM2VTS datasets. The fundamental reason lies in its non-end-to-end architecture, where the quality of synthesized results relies on the effectiveness of the encoder4editing (Tov et al., 2021) in extracting latent code. When the extracted latent code fails to capture the content of optical images effectively, severe blurring occurs in the synthesized images. Additionally, the CUFS weights provided by StyleSketch are ineffective in eliminating the influence of the background.

Tables 4 and 5 presents the FID, CF, GE and LP metrics of different methods on different datasets, where the best results are highlighted in bold. It can be observed that HCGAN in this article outperforms other comparative methods in terms of FID values across all three datasets, achieving the best performance in terms of content, global, and local aspects.

It is worth mentioning that the ability of HCGAN to refine local details depends on the capability of the local sketch refinement module to extract key local features. From Figs. 5 and 6, it can be observed that there is some blurriness in the mouth area of the synthesized images by HCGAN. The most likely reason for this is the ineffective extraction of key local features, leading to a decrease in the effectiveness of the local refinement loss. Figure 7 shows that the details in the hairline and forehead areas of the sketches synthesized by HCGAN on the SKSF-A dataset are still lacking. This inadequacy maybe since the SKSF-A dataset includes not only frontal faces but also profiles, which require more emphasis on the global-level portrayal. Under similar dataset scales, maintaining the same CUHK and other dataset settings may result in a deficiency in global-level portrayal.

Based on visual comparison and quantitative evaluation, our proposed method outperforms many existing image-to-image translation methods that use unpaired data. Specifically, our method generates synthesized sketches with more accurate and detailed local features. Whether it is the CUHK and AR datasets with fewer accessories, the XM2VTS dataset with more accessories, or the SKSF-A dataset with large angle differences, HCGAN achieves the best synthesis results, reflecting its effectiveness and robustness.