Art design integrating visual relation and affective semantics based on Convolutional Block Attention Mechanism-generative adversarial network model

Visual relationship fusion model

The article commences by providing a detailed analysis of the bias issue prevalent in existing scene graph generation algorithms. Subsequently, a comprehensive examination of the problem is undertaken, with the structure of this analysis illustrated in Fig. 1.

The model architecture proposed in this chapter focuses on enhancing the common sense-based models, specifically the Structured Neural Model (SNM) and the VTransE model. In the architecture diagram, solid thick arrows represent modules comprised of neural networks, while dashed thick arrows depict the network responsible for inferring the relationship between targets based on fused features and synthesizing target pair labels. Prior to scene graph generation, the chapter initiates a target detection pre-training process. This process utilizes Faster-R-CNN as the underlying target detector. For each image (i), the detector employs a backbone network for extracting image features. Subsequently, an RPN network with inherent anchors is utilized to learn potential target locations and their corresponding object types. Relationship sampling is then performed as part of the process.

The proposed model consists of several key modules. Firstly, there is the target context extraction module, which is responsible for capturing contextual features associated with individual targets. Next, the visual relationship modeling module extracts visual relationship features from subject-object pairs. The label prediction module performs two tasks: predicting the object categories and modeling the relationships, leveraging semantic a priori features. Finally, the relationship context feature extraction module is responsible for extracting relationship context features based on the previously extracted individual object features. The overall algorithm follows the sequence mentioned above, transforming and processing the features accordingly.

Target contextual feature extraction encompasses two main objectives: determining the object label and encoding object features within their context. To achieve effective target feature extraction and transfer target relationship knowledge, it is essential to refine the training process of target detection by back-propagating the relationship loss to the target detection network. However, the commonly used RoI pooling layer in Faster R-CNN does not employ a differentiable function for coordinate interpolation, as it relies on discrete grid partitioning of proposal regions. This characteristic makes the feature extraction layer susceptible to gradient disappearance. To address this issue, the proposed approach in this chapter replaces the RoI pooling layer with bilinear interpolation, which helps mitigate the problem of gradient vanishing. It is a smoothing function with two inputs, one input is the shape of W′ × H′ × C of the feature mapping M, the other input is the object bounding box projected onto M, the output is the object bounding box of size X*Y*C. Each input value can be efficiently interpolated from the map F in a convolutional fashion, as shown in Eq. (1). (1)${\displaystyle f_{i,j,c}=\sum _{i^{\prime }=1}^{W^{\prime }}\sum _{j^{\prime }=1}^{H^{\prime }}{M}_{i^{\prime },j^{\prime },c}k\left(i^{\prime }-G_{i,j,1}\right)k\left(j^{\prime }-G_{i,j,2}\right).}$

The visual relation modeling module is primarily utilized to extract image features of objects within an image, specifically the appearance features between pairs of objects that correspond to relations in the image. This module initiates by extracting the proposal regions associated with the target pair and subsequently computes their joint representation. Following this, feature extraction operations such as 2D convolution and batch normalization are applied to the joint representation, which are commonly employed for processing image features. Finally, a pooling operation is conducted in conjunction with bilinear interpolation to facilitate gradient backpropagation. This is illustrated in Eq. (2): (2)${\displaystyle v_e=\text{Convs}\left(\text{RoIAlign}\left(M,\text{Convs}\left(b_i\cup b_j\right)\right)\right)}$

where b_i∪b_j indicates a joint region consists of the proposal area i and the proposal area j.

Improved GAN networks

Once the scene layout of the image is obtained, the image generation process is performed using the generative network G, which is structured as shown in Fig. 2. The generative network comprises a cascaded refinement network, which is a series of convolutional refinement modules organized in a cascade fashion. Each subsequent layer of the network has twice the spatial resolution of the previous layer, enabling a coarse-to-fine image generation process. The inputs of the model are paired element-wise based on the number of channels, and then a 3x3 convolution operation is applied. The output of each submodule is upsampled using the nearest-neighbor upper-left (UL) operation before being passed to the next submodule. The output from the final module is fed into two convolutional layers to generate the final composite image output.

To better regularize the overall network and improve the learning capability of the model, especially for target samples that occur less frequently in the dataset, this article introduces a GAN module in the network’s training process. The GAN consists of a generator that utilizes a cascaded refinement network and a discriminator that employs a fully connected layer. Convolutional neural networks typically have a large number of parameters and deep network layers, which may lack local isovariance properties and suffer from reduced generalization ability with an increasing number of layers. To address this, the article incorporates capsule networks in the discriminator for reconstructing and probabilistically predicting sentences. These reconstructed sentences are expected to be semantically consistent with the images, thereby helping encode the relationship between features. The reconstructed sentences generate reward values under the discriminator and provide feedback to the generator.

The GAN-based regularizer can incorporate target information derived from the target context to ensure that the generated features align well with the target data distribution. This is achieved through a process where the target context is embedded into the GAN framework, influencing both the Generator and the Discriminator. Specifically, target context information is used to:

Guide the Generator: Adjust the Generator’s output to better match the target context’s characteristics, ensuring that synthetic data or features generated are relevant and realistic.

Refine the Discriminator: Modify the Discriminator’s criteria to focus on distinguishing between data points that are not only synthetic or real but also contextually aligned with the target.

Attention mechanism

Despite the significant advantages of neural networks in image processing, they often lack the ability to differentiate between specific attention and general attention during feature extraction. It is crucial to retain features of certain objects and regions that require focused attention. To address this limitation, this article incorporates the Convolutional Block Attention Module (CBAM), which combines both channel attention and spatial attention mechanisms. This integration allows for better feature representation and preservation of important object and region details.

The scene graph construction module in this article employs two bidirectional long short-term memory (BiLSTM) modules. These modules are utilized to model target contexts and relational contexts, generating refined pairs of target labels and relationships between targets. Specifically, it first organizes the proposal region B into a linear sequence $\left[\left({\mathrm{b}}_1,{\mathrm{f}}_1,{\mathrm{l}}_1^{\prime }\right),\dots ,\left({\mathrm{b}}_{\mathrm{n}},{\mathrm{f}}_{\mathrm{n}},{\mathrm{l}}_{\mathrm{n}}^{\prime }\right)\right]$ , where f_i is the feature extracted for each proposal region b_i is the extracted feature. ${\mathrm{l}}_1^{\prime }$ is the corresponding label, and then it is fed into the BiLSTM to obtain the object context H , where $\mathrm{H}=\left[{\mathrm{h}}_1,\dots ,{\mathrm{h}}_{\mathrm{n}}\right]$ , contains the final state of the linearized sequence elements of each proposal region in the BiLSTM, and then each element of the context is decoded according to the previously decoded labels using the LSTM as a decoder to obtain the labels of the target species. $\mathrm{L}=\left[{\mathrm{l}}_1,\dots ,{\mathrm{l}}_{\mathrm{n}}\right]$ .

For the relationship construction, another BiLSTM is used to propose regions B and object labels L as the input to the relational context C that can be represented as $\mathrm{C}=\left[{\mathrm{c}}_1,\dots ,{\mathrm{c}}_{\mathrm{n}}\right]$ , which contains the state of the final layer of each proposal region. Finally, combining the target i with the target j together with the contextual features f_i,j , the corresponding relational contexts c_i with c_j and the common knowledge information in the dataset to model the final relation r_i,j ∈ R using another LSTM for modeling, the relation R can therefore be constructed in the following way (Hameed & Garcia-Zapirain, 2020; Wei et al., 2022). (3)${\displaystyle H=\text{BiLSTM}\left({\left[f_i;W_1{l}_i\right]}_{i=1,\dots ,n}\right)}$ (4)${\displaystyle k_i={\text{LSTM}}_{\mathrm{i}}\left({\left[h_i;l_{i-1}\right]}_{i=1,\dots ,n}\right)}$ (5)${\displaystyle l_i=argmax\left(W_o{k}_i\right)\in R^{\left|G\right|}}$ (6)${\displaystyle C=\text{BiLSTM}\left({\left[f_i;W_1{l}_i\right]}_{i=1,\dots ,n}\right)}$ (7)${\displaystyle c_e=\left(W_h{c}_i\right)\circ \left(W_t{c}_j\right)\circ f_{i,j}}$ (8)${\displaystyle r_{i,j}=\text{softmax}\left(W_r{c}_e+W_{o_{i,j}}\right)}$

where W_i is the corresponding mapping matrix for each module, The network architecture incorporates a fully connected layer to implement the sentiment-guided decoding process. During decoding, the input consists of completed sentiment statements, which are learned to guide the LSTM in generating sentiment-infused descriptive statements. In this decoding phase, the output of the encoded LSTM, known as the context vector, is combined with the previously generated words through summation. This process yields the output words for the sentiment description statement.

After obtaining attention weights from both channel and spatial modules, the feature maps are scaled accordingly. This results in more informative features being passed through the network, which can improve the overall performance on tasks such as object detection and classification.

Loss function

To enhance the optimization process of the generative adversarial network (GAN) for image generation, we propose incorporating perceptual loss and adversarial loss in addition to the conventional pixel reconstruction losses. Traditional GAN training involves alternating between optimizing the discriminator D and the generator G, as described by Eqs. (9) and (10). (9)${\displaystyle L_D=E_{I\sim p_{\text{rad}}}\left[log\mathrm{D}\left(I\right)\right]}$ (10)${\displaystyle L_{\mathrm{G}}=E_{\hat{i}{p}_{\mathrm{G}}}\left[log\left(1-\mathrm{D}\left(\hat{I}\right)\right)\right]+\lambda L_i}$

where λ is the adjustable parameter, for the loss of the generative network, for better gradient performance, we maximize $log\mathrm{D}\left(\mathrm{G}\left(\mathrm{z}\mid {\mathrm{S}}^{\text{layout}}\right)\right)$. For the loss of the reconstructed image comparing the reconstructed image with the original image L_i, the distance between the real image I and the synthetic image $\hat{I}$ is calculated, which can be expressed as ${\mathrm{L}}_{\mathrm{i}}=\parallel \mathrm{I}-\hat{I}{\parallel }^1$.

To further refine the quality of the generated images, we introduce perceptual loss and adversarial loss into the optimization process. Perceptual loss focuses on capturing high-level features and textures by comparing feature maps extracted from a pretrained network, such as VGG. This loss function operates at the feature level rather than pixel level, ensuring that the generated images preserve high-level details similar to the reference images. The perceptual loss L_p can be formulated as: (11)${\displaystyle L_p=\sum _{l\in \text{layers}}{∥{\varphi }_l\left(I\right)-{\varphi }_l\left(\hat{I}\right)∥}_2}$

where ϕ_l represents the feature maps extracted from layer l of the pretrained network.

Adversarial loss, on the other hand, enhances realism by training the generator and discriminator in tandem. The adversarial loss encourages the generator to produce images that are indistinguishable from real images by the discriminator. The total adversarial loss L_adv can be expressed as: (12)${\displaystyle L_{adv}={\mathbb{E}}_{I\sim p_G}\left[logD\left(\hat{I}\right)\right].}$

By combining these additinal losses, the updated total loss function for the generator G is: (13)${\displaystyle L_G={\mathbb{E}}_{I\sim p_G}\left[log\left(1-D\left(\hat{I}\right)\right)\right]+{\lambda }_1{L}_i+{\lambda }_2{L}_p}$

where λ₁ and λ₂ are adjustable parameters that control the influence of the reconstruction loss and perceptual loss, respectively. Incorporating perceptual loss and adversarial loss not only improves the fidelity of the generated images but also enhances their overall visual quality and feature accuracy. This refined approach effectively combines the strengths of pixel-based and feature-based losses, leading to more realistic and high-quality image generation.

Formatting of mathematical components

The MSCOCO dataset (DOI: 10.5281/zenodo.7517539) was utilized in the experiment, which includes a minimum of five captions for each image. For the experimental setup, the longest caption in terms of length was selected as the reference. The sentence length was fixed at 40, and any captions shorter than 40 were padded with “<pad>”. The training process involved starting the captions with “<start>” and ending them with “<end>”. Data augmentation techniques such as “rotation” and “hue transformation” were applied to enhance the dataset. For the augmented dataset, 89% of the images were used for training, while the remaining 11% were used for testing. Additionally, the Senticap corpus, which provides objective and descriptive sentiment-based image utterances derived from MSCOCO, was incorporated. The Senticap corpus contains 1,027 adjective-noun pairs for positive emotions and 436 adjective-noun pairs for negative emotions.

In this article, the evaluation of text sentiment conversion was measured using the accuracy (ACC) metric. The preservation of textual content between the generated text and the original text was assessed using the content preservation rate (CON). The readability of the generated text was evaluated using BELU, where a higher BLEU score indicates better readability and a smaller difference from the original text.

Results and discussion

Model comparison

In this article, several models are used for comparison with the proposed approach. These models include:

VCTree (Tang et al., 2019): VCTree utilizes a scoring function to compute the task-dependent validity between each pair of objects, forming a tree structure. It employs a bidirectional tree-like LSTM to encode visual relations and a task-specific model for decoding.

IMP (Xu et al., 2017): IMP addresses the scene graph inference problem using a standard recurrent neural network (RNN). It improves prediction performance by incorporating contextual message passing.

SNM+TDE (Tang et al., 2020): SNM+TDE generates the final inference by subtracting the probability distribution of direct inferences from the scene graph model from the probability distribution of counterfactual inferences from the same model.

VTransE (Xu et al., 2020): VTransE is the pioneering approach that applies knowledge representation learning concepts to scene graph generation. It represents relations as embeddings of subject and object differences.

These models serve as baselines to evaluate and compare the performance of the proposed approach in the article. The training process of different models and their accuracy comparison results are presented in Fig. 3. It can be observed that VCTree, IMP, VTransE, and SNM+TDE models reach convergence in approximately 25-30 rounds for VCTree and IMP, and 35-40 rounds for SNM+TDE. In contrast, the proposed model in this article achieves convergence in approximately 10 rounds. Moreover, the accuracy of each data group is significantly improved compared to the other models. This suggests that the proposed model performs better in terms of convergence speed and accuracy on the given dataset.

In addition, performance comparisons of the different models are shown in Figs. 4 and 5.

The proposed model demonstrates superior performance compared to the baseline models in terms of accuracy, BLEU score, and text preservation rate. With an accuracy of 95.42% and a BLEU score of 16.79, the proposed model exhibits enhanced capabilities in visual semantic mining. In contrast, the baseline models achieve high accuracy rates exceeding 85%, but their BLEU scores are significantly lower. This discrepancy can be attributed to training conditions that lack an adequate parallel corpus, causing these models to focus on isolating semantic feature words in the text without considering the coherence of semantic information. Consequently, the generated text from these models outputs sentences with the target sentiment but lacks overall coherence. In contrast, the proposed model employs a self-attentive mechanism to identify sentiment words with extensive sentiment lexicons in the sentences, storing them in the memory network. This approach enables the model to better consider the semantic context, thereby enhancing the text preservation rate and readability of the generated text.

Figure 6 portrays a fortuitous illustration of sentence generation for each model procured through a specific modeling approach. The model introduced in this manuscript showcases a heightened level of semantic cohesion, while exhibiting remarkable similarity to the original text.

The adverse depictions encompass a range of negative human emotions, such as the chilling air, desolate mountain, and solitary street, all characterized by negative adjectives. Conversely, the affirmative portrayals involve pleasant weather, bustling streets, and incorporate neutral terms alongside positive and optimistic adjectives. The VCTree model represents sentence vectors through training that eschews the use of parallel corpora and unrelated content, thereby ensuring the preservation of semantic style within the text. Meanwhile, the SNM+TDE model enhances text fidelity by eliminating specific words that signify textual features, thereby diverting the model’s focus from generating sentiment akin to the original text. Furthermore, if the sentiment expressions within the text are cryptic, these models lack the ability to effectively extract the semantic information required to train samples that accurately mirror the original text. The experimental outcomes not only explicate the image content but also manifest emotional nuances, encompassing both negative and positive human emotions. This serves as compelling evidence of the model’s substantial improvement in terms of emotional richness and performance.

Discussion

Learning with contextual features tends to outperform non-bilinear pooling in various applications. Non-bilinear pooling, which incorporates image features, often negatively impacts the performance of networks such as SNM and VTransE. This is because non-bilinear pooling separates the “style” component of image features from the “content” component of contextual features, which include crucial information for modeling object relationships, such as object labels. The combination of these two components forms the basis for determining object relationships.

It is important to note that the same content can be expressed in different styles. For example, different images depicting a “man near a bird” may have similar contextual features, as they both contain elements of “bird” and ”man”, indicating a “close” relationship. However, the bird may not always be in a fixed position relative to the person, and it may not always be depicted flying in the sky, resulting in variations in image features. Non-bilinear pooling approaches disregard the correspondence between these factors, leading to reduced effectiveness compared to baseline approaches that solely focus on content.

In the field of art design, accurate generation and expression of emotional semantics can greatly enhance the interactive experience between viewers and artworks. It bridges the gap between the viewer and the artwork, enabling a deeper understanding and engagement with the artistic piece. The innovation of artificial intelligence image creation lies in the ability to deviate from the original style within a specific range, while adhering to art identification standards and learning from deviations. This involves pushing the boundaries of established artistic styles within a given art category.

By leveraging big data to build relational networks and employing data analysis models and deep learning models developed by data scientists, valuable knowledge can be extracted from existing data. These advanced techniques enable the exploration of new possibilities in art creation and contribute to the evolution and advancement of the field.