A convolutional neural networks based approach for clustering of emotional elements in art design

Visual communication

As a professional field of modern new media form design, visual communication design studies the cultural awareness issues involved in this profession, how to draw on traditional culture for innovative creativity, etc. It solves misunderstandings and misinterpretations in the current understanding and avoids paradigm design brought by superficial cognition. Based on the higher innovation expectations and calls for visual communication design in the information age, design form innovation must be adapted to the progress of society and the development of the times. Visual culture is generally classified as socio-cultural science, which integrates the results of cultural studies and visual studies. Its scope gradually expands to comprehensive theoretical cross-disciplinary and multi-disciplinary research systems, such as social culture, art, philosophy, communication, and design (Kujur & Singh, 2020). Mirzoeff, a prominent scholar in the realm of visual culture, expounds that visual culture emerges as an inherently interdisciplinary field. Echoing Roland Barthes’ sentiment, Mirzoeff highlights the notion that cross-disciplinary endeavors involve the identification of a central theme, subsequently orchestrating the convergence of two or three distinct disciplines around it. This interdisciplinary synergy aims to generate an innovative intellectual construct, one that transcends the boundaries of individual disciplines (Mirzoeff, 1999). As a result, the concept of visual communication has progressively evolved into an indispensable framework within the domains of art and design. Vigoroso, Caffaro & Cavallo (2020) introduces visual communication and design principles, including perception, composition, typography, colour, and rhetoric. It also explores how graphical images are used in various contexts, such as advertising, journalism, art, and film. Bringhurst (2004) discusses typography’s historical and cultural significance and includes examples of effective design from different historical periods and cultures. Henderson (1991) present a framework for understanding how graphic desigfn affects reading based on empirical. It explores the effects of typography, layout, colour, and imagery on reading speed, comprehension, and memory. Houts et al. (2006) reviews the research on the design of visualizations for effective communication, covering topics such as perception comprehension. The study discusses the factors that affect the effectiveness of visualizations, such as visual encoding, structure, and interaction, and offers recommendations for designing compelling visualizations (Houts et al., 2006). Richardson, Drexler & Delparte (2014) reviews the literature on graphic design principles for e-learning, including topics such as typography, colour, layout, and multimedia. The study discusses the implications of these principles for the design of effective e-learning materials and makes recommendations for integrating visual design into the e-learning design process (Richardson, Drexler & Delparte, 2014).

Lastly, the incorporation of domain-specific knowledge or constraints into deep learning models can be explored. Design principles and expert knowledge play a vital role in poster design, and leveraging such information during the clustering process could lead to more meaningful and context-aware results. Integrating domain-specific knowledge with deep learning models could enhance the quality and relevance of the clustering outcomes. Through the above study, it is easy to see that as visual communication theory continues to expand, people are applying it to various fields of art design, no longer just limited to the creation of traditional media mediums. We can obtain more information by refining the various elements in typographic design.

Sentiment element clustering

As artificial intelligence technology advances, the desire for abstract features has grown beyond the traditional visual elements. Researchers must now consider the potential of arranging and combining features of a single part to create new abstract features. Therefore, it is crucial to analyze the emotions expressed in art design using deep learning techniques, and deep learning has a wide range of applications in such research. The deep self-encoder model proposed by Hinton & Salakhutdinov (2006) is a feature extraction model with excellent performance; the algorithm maps high-dimensional data to low-dimensional space by an encoder and then recovers the original data by the decoder to reconstruct the low-dimensional space data features in an unsupervised training manner. When the model converges, the encoder can achieve the role of feature extraction and data dimensionality reduction. Vincent et al. (2010) proposed a stacked denoising autoencoder model to obtain a more robust algorithm for stability. The model uses data with noise instead of the original data to train the autoencoders, and the resulting model has more substantial generalization and noise immunity than the conventional autoencoders. Since deep models have more vital feature mining ability than shallow models, the stack denoising self-encoder model increases the depth of the model by cascading multiple denoising self-encoders. In addition, the stack-denoising self-encoder uses a layer-by-layer training method to alleviate the problem of a complex convergence of the depth model.

The Stochastic Neighbor Embedding (t-SNE) algorithm was proposed by Der Maaten & Hinton (2008) to address the problem that traditional distance metric algorithms have difficulty characterizing the differences between high-dimensional data. The algorithm offers a novel distance metric instead of Euclidean distance. First, a probability vector is constructed for each category the data point belongs to based on the distance from the data point to each centre of mass. Then a probability vector is built similarly in the feature space after data dimensionality reduction. The clustering model is optimized by minimizing the two probability vectors’ relative entropy (Kullback–Leibler Divergence, KL). Xie, Girshick & Farhadi (2016) proposed a combined deep embedded clustering (DEC) model combining stacked denoising self-encoder and t-SNE algorithm. The model obtains an encoder capable of feature extraction by greedily training a stack denoising self-encoder layer by layer. Then, the encoder parameters are fine-tuned to optimize the loss of t-SNE and complete the clustering. Guo et al. (2017) proposed optimized deep embedded clustering based on the model IDEC model, which uses the loss of the self-encoder and the loss of clustering on feature data as multiple loss functions to train the network, thus preventing the fine-tuned self-encoder network from the original self-encoder feature extraction space from being corrupted.

While deep learning methods have shown immense potential in various applications, including poster design clustering, there are still certain aspects that can be improved compared to the method proposed in this article.

One aspect that requires attention is the computational complexity associated with deep learning models. Deep clustering algorithms often involve training large-scale neural networks, which can be computationally expensive and time-consuming. Improving the efficiency and scalability of these models is crucial to enable faster data clustering, particularly in scenarios where large datasets or real-time clustering are involved. Another area for improvement lies in the interpretability of deep learning models. Deep clustering methods often rely on complex architectures and numerous parameters, making it challenging to interpret and understand the underlying reasoning behind the clustering results. Enhancing the interpretability of deep learning models would provide designers and researchers with valuable insights into the clustering process and help them make more informed decisions.

Furthermore, the robustness and generalization capabilities of deep learning models can be improved. It is essential to ensure that the proposed clustering framework performs consistently well on diverse datasets and is not overly sensitive to variations in input data or minor changes in the design elements. Robust models would provide reliable and stable clustering results across different scenarios and design styles.

The above research shows that deep learning methods have made good research progress in clustering. For the more abstract emotional elements of the art design, the use of deep clustering to complete the given poster with fast data clustering is of great significance for future intelligence and rapid innovation.

Image processing under convolutional neural network

CNN is a deep learning model commonly used for image recognition and computer vision tasks (Shi et al., 2022). It progressively extracts feature information from input images through different hierarchies such as convolutional layers, pooling layers and fully connected layers to achieve tasks such as classification, localization and recognition of images (Bhatt et al., 2021).

The core of CNN is the convolution operation, the basic idea of which is to slide a convolution kernel (also known as a filter) over a picture, weigh it, and sum it with the corresponding pixel values in the image to obtain a new feature map. The convolution operation helps the network to capture the local features in the picture, thus enabling feature extraction for the whole picture. Specifically, the convolution operation can be expressed as the following equation (Lin et al., 2022):

(1)${\displaystyle {\mathrm{y}}_{\mathrm{i},\mathrm{j}}={\sum }_{\mathrm{m}=1}^{\mathrm{M}}{\sum }_{\mathrm{n}=1}^{\mathrm{N}}{\mathrm{w}}_{\mathrm{m},\mathrm{n}}{\mathrm{x}}_{\mathrm{i}+\mathrm{m}-1,\mathrm{j}+\mathrm{n}-1}+\mathrm{b}}$

where denotes the pixel values of row y_i,j i and column j in the new feature map obtained after convolution; ^{x_{i+m−1,j+n−1}} denotes the pixel values of row i + m − 1 and column j + n − 1 in the original image; ^w_m,n denotes the weight values of row m and column n in the convolution kernel; and b represents the bias term.

To reduce the dimensionality and the number of parameters of the feature map, CNNs usually use pooling operations. The pooling operation can count the pixel values of each local region in the feature map to obtain a new pooled feature map. The common pooling operations are maximum and average pooling, as in Eqs. (2) and (3):

(2)${\displaystyle y_{i,j}=max\left(x_{i+m-1,j+n-1}\right)}$

(3)${\displaystyle y_{i,j}=\frac{1}{MN}{\sum }_{m=1}^M{\sum }_{n=1}^N{x}_{i+m-1,j+n-1}}$

where y_i,j denotes the pixel values of the row i and column j in the new feature map obtained after pooling; x_{i+m−1,j+n−1} indicates the pixel values of the row i + m − 1 and column j + n − 1 in the original feature map; M and N denote the size of the pooled region. By continuously using convolution and pooling operations, the CNN can gradually extract the high-level feature information in the input image, thus achieving tasks such as classification and recognition.

To ensure the robustness of the convolution operation and the ability to accurately detect targets within images, even if they undergo translation, downsampling using pooling layers becomes essential. The utilization of pooling layers serves the purpose of reducing the size of the feature map.

The two commonly employed pooling operations are maximum pooling and average pooling. Maximum pooling selects the maximum value within each rectangular region of the feature map as the output, while average pooling calculates the average value within the region. By applying pooling operations, the spatial information is condensed while retaining the most salient features.

The convolutional and pooling layers are arranged in an alternating fashion and can be stacked multiple times, giving rise to deep networks. The depth of CNN allows for the learning of increasingly complex features, enhancing their ability to recognize and classify images effectively. As the network deepens, it becomes capable of capturing hierarchical representations, enabling it to discern intricate patterns and subtle variations in the input data. At the topmost layer of the network, a fully connected layer typically consolidates the learned elements from previous layers. This layer integrates the extracted features and produces the final classification output. By combining the representations learned throughout the network, the fully connected layer captures the high-level information necessary for making accurate predictions or classifications. The hierarchical arrangement of convolutional and pooling layers, coupled with the final fully connected layer, forms a comprehensive deep learning framework. This architecture empowers CNN to learn intricate and abstract features, leading to improved image recognition and classification performance. The ability to capture complex visual patterns and generalize from learned features contributes to the success of deep CNN in various computer vision tasks.

In this article, we use CNN to extract sentiment element features and complete the clustering analysis of sentiment elements based on the generative model.

CNN-based clustering of VAE sentiment elements

While traditional clustering methods often encounter many difficulties when dealing with large-scale high-dimensional data, deep learning-based clustering methods are becoming increasingly popular due to their high-dimensional nature. One of the deep learning-based clustering methods is variational autoencoder (VAE), a generative model that enables efficient data compression and image reconstruction through probabilistic coding and decoding techniques (Hu et al., 2023). VAE is a self-coding neural network that maps input data into a distribution of potential spaces by an encoder, which a decoder then samples from that distribution to generate data with similar features. In this process, VAE learns a low-dimensional representation of the data by minimizing the reconstruction error and the regularization term in the potential space, which makes the clustering analysis more efficient. The process of cluster analysis using VAE is roughly divided into two steps: first, a low-dimensional representation is obtained by using VAE for dimensionality reduction. Then, the clustering algorithm is used to cluster the reduced-dimensional data to get the clustering results of the emotional elements in art design. Due to the efficiency and excellent performance of VAE, it has been widely used in image clustering, audio clustering, text clustering, etc. (Anvekar et al., 2022).

VAE aims to minimize the empirical evidence lower bound (ELBO) which consists of two components: reconstruction loss and a potential spatial regularization term. The reconstruction loss represents the reconstruction error on the original data, while the potential spatial regularization term can constrain the distribution of potential variables by increasing the KL scatter. Specifically, VAE maps the input data x into a mean vector µand a standard deviation vector σ and generates the latent variables by randomly sampling z from a Gaussian distribution N(μ, σ). Thus, the training objective of VAE can be expressed as the following equation:

(4)${\displaystyle \mathfrak{L}={\mathbb{E}}_{Z\sim q\left(z\mid x\right)}\left[logp\left(x\mid z\right)\right]-D_{KL}\left(q\left(z\mid x\right)\parallel p\left(z\right)\right)}$

where p(x∣z) denotes the decoder’s generative distribution, q(z∣x) denotes the encoder’s latent variable distribution, p(z) denotes the prior distribution, and D_KL denotes the KL scatter, which is calculated as shown in Eq. (5):

(5)${\displaystyle D_{KL}=-\frac{1}{2}\sum i=1^n\left(1+log\left({\sigma }_{i}2\right)-{\mu }_{i}2-{\sigma }_{i}2-e{z}_i\right)}$

where e^z_i is an empirical parameter that is used to limit the range of potential space vectors and is usually set to a small value. During the training process, VAE uses a backpropagation algorithm to calculate the gradient of the loss function and updates the parameters in the network based on the slope. Specifically, VAE uses the stochastic gradient descent (SGD) algorithm or its variants for optimization.

The calculation process of backpropagation mainly involves the solution of the error in each stage, where the output layer error is shown in Eq. (6):

(6)${\displaystyle {\delta }_j^L=\frac{\partial C}{\partial a_j^L}{\sigma }^{\prime }\left(z_j^L\right)}$

where C is the loss function, $a_j^L$ is the activation value of the ^j neuron in the output layer, $z_j^L$ is the weighted input of the ^j neuron in the output layer, and σ′ is the derivative of the activation function. The error of the hidden layer is then.

(7)${\displaystyle {\delta }_{j}l={\sigma }^{\prime }\left(z_j^l\right){\sum }_k{\delta }_k^{l+1}{w}_{jk}}$

where l it indicates the number of the current layer, ^w_jk is the weight between the neuron in the kl + 1 layer and the neuron of ^j in the ^l layer, and ${\delta }_k^{l+1}$ is the error of the neuron k in the l + 1 layer. Based on this, we can calculate the bias gradient and weight gradient to complete the backpropagation calculation of the model error; for reducing the model error, the bias gradient and weight gradient are calculated as shown in Eqs. (8) and (9):

(8)${\displaystyle \frac{\partial C}{\partial b_{j}l}={\delta }_{j}l}$

(9)${\displaystyle \frac{\partial C}{\partial w_{jk}l}=a_{j}l{\delta }_k^{l+1}}$

Therefore, the classification process of emotional elements in art design using the VAE method in this article is shown in the algorithm:

The poster design framework with the ability to cluster emotional elements proposed in this article is shown in Fig. 1.

This article delves into the exploration of abstract emotional elements within the realm of visual communication art design. To begin with, a key focus is placed on leveraging the capabilities of CNN to accomplish the crucial task of feature extraction from the informational content of poster images. By employing CNN, the emotional elements embedded within the visual designs are effectively captured and represented. Subsequently, these extracted features serve as the foundation for conducting a comprehensive cluster analysis of the emotional elements, employing the powerful VAE methodology. Through this approach, a deeper understanding of the underlying emotional characteristics and patterns in the poster designs is achieved, paving the way for innovative insights and advancements in the field of visual communication art design.

The effect of sentiment clustering

To generate images, we conducted data clustering analysis based on the data feature points, and different colour data represent the related emotional performance. The clustered results are shown in Fig. 2.

Considering the privacy of the data and the copyright characteristics, we only show the feature clustering results under the two classes of features extracted from the data. It can be seen that after the VAE method, the three classes of components can be distinguished, and to evaluate the effect of clustering, we calculated the silhouette coefficient, the results of which are shown in Fig. 3.

According to the results in Fig. 3, the CNN-VAE framework used in this article has a more balanced performance in the clustering of three types of emotions. The silhouette coefficient is over 0.7, which indicates that its clustering effect is better. Meanwhile, we tested its precision in the subsequent experiments to illustrate the clustering effect more intuitively.

Comparison of different clustering methods

To verify the clustering effect, we selected three more classical clustering algorithms in clustering methods for comparison, namely K-means, FCM and SOM, and the accurate label comparison accuracy under different ways is shown in Fig. 4.

From Fig. 4, we can find that the accuracy is significantly higher than that of the traditional method after clustering, and the recognition of the three types of emotions is also more balanced. Hence, the model has great potential for future applications.

Practical analysis of clustering effects

To better verify the actual effect of clustering, we made artificial poster modifications in the real test by adding the typical emotional elements in 4.1 to the designed poster, whose accuracy after clustering is shown in Fig. 5.

Through the data in Fig. 5, we can find that the proposed framework achieves good accuracy in all three categories of sentiment clustering, in which the success rate of clustering posters involving negative elements reaches 87.9%, and its clustering ability is fast and more accurate for art designs with specific characteristics.

Framework operation efficiency testing

Because in practical applications, designs containing elements of a particular class are often analyzed quickly, it is crucial to cluster a single kind of sentiment element. In this article, a system operation efficiency test for a single sentiment clustering was conducted, and the results are shown in Fig. 6. Execution Time: This metric measures the time taken by the framework to execute a particular task or operation. It can be calculated by measuring the time difference between the start and end of the operation.

During the evaluation process, when focusing solely on a single sentiment element, the model consistently demonstrated impressive efficiency rates surpassing 85% across all test cases. This noteworthy performance attests to the model’s effectiveness and efficiency when applied specifically to a single sentiment element. Such high efficiency holds great significance for the widespread application of the model in future endeavors. By excelling in the analysis of individual sentiment elements, the model’s capabilities align with the demand for targeted and specialized sentiment analysis tasks. The ability to efficiently process and analyze sentiments in isolation opens up a multitude of possibilities for diverse applications, spanning various domains such as market research, social media analytics, and sentiment-driven decision-making processes. As a result, the model’s exceptional efficiency in handling single sentiment elements represents a valuable contribution towards advancing sentiment analysis techniques and their practical deployment across a wide array of real-world scenarios.