GAN inversion and shifting: recommending product modifications to sellers for better user preference

User preference

Visual recommendation systems utilize visual features extracted from an item image to model user preference. Kang et al. (2017) propose an end-to-end learnable (all parameters trained in one go) approach called deep-visually aware bayesian personalized ranking that learns dataset-specific visual features from scratch using a convolutional neural network (CNN) while focusing on an implicit feedback problem. In the context of implicit feedback, the only available information is whether a user has clicked on an item. Consequently, the recommendation problem reduces to ranking previously observed (clicked) items higher than unobserved (unclicked) items. The DVBPR (Kang et al., 2017) method formally describes the problem as follows: Given a dataset D where I denotes the set of items and U denotes the set of users, we need to rank items set $I\setminus I_u^{+}$ with which a given user u ∈ U has not interacted with. This ranking is done using the preference score x_u,i which is the preference of user u ∈ U towards item $i\in I_u^{+}$ using the item image X_i. This is mathematically represented by Kang et al. (2017) using the following equation: (1)${\displaystyle x_{u,i}={\theta }_u^T\varphi \left(X_i\right),}$

Here θ represents the user embedding and ϕ represents the weights of a CNN architecture which are learned using the Bayesian personalized ranking (BPR) optimization framework (Rendle et al., 2012). BPR optimises the ranking using triplets u, i, j ∈ D, where: (2)${\displaystyle D=\left\{\left(u,i,j\right)|u\in U\wedge i\in I_u^{+}\wedge j\in I\setminus I_u^{+}\right\},}$

Here $i\in I_u^{+}$ is an item observed by user u ∈ U and $j\in I\setminus I_u^{+}$ is an unobserved item. Thus ideally, the preference score (Eq. (1)) of item i should be more than item j, hence BPR objective function is defined as following: (3)${\displaystyle max\sum _{\left(u,i,j\right)\in D}\left(ln\left(\sigma \left(x_{uij}\right)\right)-{\lambda }_{\Theta }\left|\right|\Theta |{|}^2\right),}$

Here σ(⋅) is the sigmoid function, λ_Θ||Θ||² as the regularization term with λ_Θ as the regularization hyper-parameter. The Θ is the set of all the trainable weights of a neural network, and the term x_uij is the preference score difference between item i and item j for a given user u and is mathematically represented as follows: (4)${\displaystyle x_{uij}=x_{ui}-x_{uj}.}$

Perceptual similarity (LPIPS)

Given the seller-oriented focus of our work, we employ perceptual similarity as a key metric in our methodology. The learned perceptual image patch similarity (LPIPS) metric, as discussed by Zhang et al. (2018), is foundational to our approach. LPIPS is based on the notion of human perceptual similarity, reflecting how humans assess the similarity between image patches. This metric is critical for evaluating and improving the alignment of fashion items with user preferences from a seller’s perspective. Formally, the LPIPS metric is defined as the l₂ distance between activations extracted from a stack of layers. The author uses activations (w) obtained from different layers (l) of CNN as it provides a better representational space for various tasks. LPIPS is mathematically defined as: (5)${\displaystyle d\left(x,x_0\right)=\sum _l\frac{1}{H_l{W}_l}\sum _{h,w}\left|\right|w_l\odot \left({\hat{y}}_{hw}^l-{\hat{y}}_{0hw}^l\right)|{|}_2^2,}$

Here, for two image patches x and x₀, we calculate the spatially averaged (H∗W), l₂ distance aggregated over channel (C) between the ${\hat{y}}^l$, ${\hat{y}}_0^l\in R^{H_l\ast W_l\ast X_l}$. The ${\hat{y}}^l$ and ${\hat{y}}_0^l$ are obtained from unit normalized internal activations scaled (using Hadamard product [⊙]) channel-wise by w_l ∈ R^C_l extracted from different layers (l) of a network F.

Overview

Our method builds upon a pre-trained generative adversarial network (GAN) model (Goodfellow et al., 2014; Karras et al., 2017; Brock, Donahue & Simonyan, 2018). A GAN consists of two components: a generator and a discriminator. These models improve through adversarial training, where the generator aims to produce images that the discriminator cannot distinguish from real images, thereby maximizing the discriminator’s loss. Conversely, the discriminator learns to accurately classify generated images as fake and real images as real, minimizing its classification loss. This adversarial process enables the generator to create images that do not exist in the original dataset, showcasing the GAN setup’s ability to generate novel and realistic images.

This generative ability leads to their utilization in applications like image inpainting, facial attribute modification, and novel clothing generation. Kang et al. (2017) also utilizes the power of GANs to generate images for a given class of clothing to have a high preference score for a given user. This optimization process is mathematically expressed as: (6)${\displaystyle \hat{\delta }\left(u,c\right)=\underset{e\in G\left(\cdot ,c\right)}{argmax}{\hat{x}}_{u,e},}$ (7)${\displaystyle {\hat{x}}_{u,e}={\theta }_u^T\varphi \left[G\left(z,c\right)\right]-\eta {\left[D_c\left(G\left(z,c\right)\right)-1\right]}^2,}$

Here, Eq. (7) is the loss that contains the user preference score and discriminator loss of whether the generated image is real or fake. Thus, the author tries to optimize the generator such that it efficiently explores the latent data manifold for vectors corresponding to a given class leading to real and highly preferred item images. However, their work had a few limitations. First, on the methodology side, the actual generated images were of inferior quality. The reason for such low-quality images is that their problem formulation is much harder to solve because a GAN has to map a latent vector with both a given class and a given user to generate images. Also, the optimized objective is excessively biased toward maximizing user preference instead of image quality, which introduces noises that fool the recommendation models into believing that the image will be preferred by the user (Brown et al., 2017a; Liu et al., 2019). Moreover, at a given time the Kang et al. (2017) generator would only generate images that belong to a given class basis a user’s preference. This is hardly a situation in a real-life scenario, where a seller would want how he can make changes to his current products such that the modified product is ranked higher in the user preference list for a set of users (instead of a single user), doing so in a way that the modified image looks realistic.

These drawbacks motivated us to propose and build a framework that is more seller/manufacturer oriented allowing them to gain insights from their item consumer’s preferences, and generalizable to any item by suggesting novel changes considering the item’s current image. To do this, we utilize a pre-trained GAN architecture and explore concepts from both GAN inversion theory (Abdal, Qin & Wonka, 2019; Abdal, Qin & Wonka, 2020; Bau et al., 2020) and GAN latent disentanglement theory (Voynov & Babenko, 2020). GAN inversion deals with accurately back-mapping a given latent vector to an image, and GAN latent disentanglement refers to the exploration of useful directions within a GAN’s latent space. One of the main reasons for us to use a pre-trained GAN architecture is to leverage its ability to generate high-quality/resolution images and solve our problem of generating realistic fashion images. Since we are using a GAN model that only takes in a latent vector as input, hence for a given item image, we need to find where this item is situated in the latent space of the GAN model which is very similar to the reverse diffusion. Thus, we build a process depicted in Fig. 2 where a latent vector is optimized to minimize a combination of perceptual loss defined in Eq. (5) and mean-square-error between the image generated from the optimized latent vector and the actual item image. The perceptual similarity loss helps in learning how two images are perceptually similar, and the mean-squared error loss helps in making the generated image look closer to the actual item image digitally.

After obtaining the latent vector, we fix its position and explore the surrounding latent space within a defined radius. This process involves optimizing a set of shifting vectors that when added to the base latent vector, produce images that are physically proximate to the base image yet exhibit aesthetic diversity. In addition to this, objective of this step is to maximize the user preference score while simultaneously minimizing diversity loss which indicates dissimilarity among shifting vectors, and shift prediction loss. To achieve this, our method utilizes a model composed of a ResNet18 feature extractor and several linear layers (as illustrated in Fig. 3) to calculate diversity loss and shift prediction. These metrics ensure the maintenance of uniqueness across the entire batch of shifting latent vectors.

By combining both components of base latent vector search and shifting vectors, our methodology aims to produce images closely resembling the given item image while catering to the preferences of user groups, thus enhancing its generalizability. Furthermore, our approach can also be customized for generating images at a global level by using randomly sampling base latent vectors, thereby bypassing the need for a seller image as the base. Similarly, by adjusting shifting vectors across a range of users rather than a single user, it becomes possible to identify items preferred by a broader user base without knowing about any item. These customizable enhancements improves the versatility and applicability of our methodology across diverse scenarios.

 
_______________________ 
 Algorithm 1: GIGS Training                                                    _________ 
    Initialise: Generator G(⋅), Recommendation Feature Extractor ϕ(⋅), 
      Shift predictor Θs(⋅), dataset S, training iterations T, batch size for 
      shifting vector n, Targets (Y ∈{0,...,n − 1}), base latent vector 
      learning rate α1, shifting vector learning rate α2, search radius ϵ; 
    for each data point {Ui,Xi}∈ S do 
        Sample: Base latent vector z ∈ N(0,1); 
   for iterations = 1 to T do 
       ˆ Xz ← G(z); // Base Image 
  Llpips ← LPIPS(Xi, ˆ Xz); 
   Lmse ← (Xi − ˆ Xz)2); 
   Linv ← Llpips + Lmse; // Inversion Loss 
  z ← z − α1 ⋅∇zLinv; // Backpropagation 
  z ← z−μz 
  σz  ; // Normalization 
  Sample: Shifting vector batch {k}n ∈ N(0,1); 
   for iterations = 1 to T do 
       k ← min(max(k,−ϵ),ϵ); // Clipping Shifting Vectors 
  kz ← z + k; // Shifting Base Latent Vector 
  kz ← kz−μkz 
  σkz   ; // Normalization 
   ˆ Xk ← G(kz); // Generating Shifted Images 
  Lpref ← θTU 
iϕ[Xk]; // Preference Score 
  Pshift,Pclass ← Θs(Xk); 
   Lclass ← Lce(Pclass,Y ); 
   Lshift ← Lmae(Pshift,k); 
   Ls ← Lclass + Lshift − Lpref; // Shifting Loss 
  k ← k − α2 ⋅ 1 n∑(∇ 
kLs); // Backpropagation 
  Θs ← Θs − α3 ⋅ 1 
n ∑ 
   (∇ΘsLs); // Backpropagation 
  Obtain:   ˆ Xk // Shifted Images    

Algorithm

To train our proposed methodology we use Algorithm 1 where we begin by initializing the pre-trained generator network G(⋅), Feature extractor ϕ(⋅) obtained from recommendation model, Shift predictor Θ_s(⋅), dataset S, no. of iterations T to run the training process, no. of output images n, Targets Y ∈ {0, …, n − 1}, learning rate of base latent vector α₁, learning rate of shifting vector α₂, and learning rate of Shift predictor α₃

We then run our method for each randomly sampled user U_i and a given item image X_i. We first begin by running the GAN inversion process where we first initialize base latent vector z with normal distribution N(0, 1). Then for each iteration in range 1 to T, we first pass z to the generator G(⋅) and obtain the image ${\hat{X}}_z$ corresponding to z. Afterward, we calculate perceptual similarity (L_lpips) and mean-squared error loss (L_mse) and optimize z to minimize the summation of both of these losses. After the optimization, the optimized z may no longer be within the normal distribution. Hence we normalize it to have a mean of 0 and a standard deviation of 1.

After the first loop is finished, we obtain our final base latent vector z. We build upon z to get a batch of shifting vectors {k}ⁿ, when added to z results, in images with high user preference. To do this we begin by initialising {k}ⁿ with normal distribution N(0, 1). Then for each iteration in the range of 1 to T, we clamp the shifting vector to between a search radius of ϵ. This clamping helps our process to only search for images that are somewhat similar to the image obtained using z. After this, we shift our base latent vector z with batch k to obtain a batch of shifted latent vectors {k_z}. Now we normalize the shifted latent vectors to have a normal distribution. This normalized variant of the shifted latent vector is then passed down to the generator G(⋅) to obtain shifted images ${\hat{X}}_k$, which we use to calculate the preference score L_pref for the user U_i. To keep all the vectors present in shifting vector batch {k}ⁿ unique, we utilize a shift predictor Θ_s(⋅) that learns to predict the {k}ⁿ as P_shift and the probability P_class with which each image in the batch X_k lies in category Y ∈ {0, …, n − 1}. This category Y is obtained using the batch size of shifting vector {k}ⁿ. These values are then used to calculate the cross entropy loss L_ce and mean absolute error L_mae. Finally, the batch of shifting vector {k}ⁿ and shift predictor Θ_s(⋅) is optimized to minimize L_s, which is the summation of L_ce and L_mae from which we subtract L_pref. After completing all the iterations, we obtain the new shifted images (${\hat{X}}_k$) that are closer to the base image and have high user preference.

Dataset

We train, test, and compare our methodology on the Amazon fashion dataset (Kang et al., 2017; McAuley et al., 2015), which is a comprehensive real-world dataset with a large number of user-item interactions. Moreover, this dataset contains a large variety of item images mapped to 6 different classes (such as shirts, jackets, trousers, shoes, sandals, etc.) for both genders (male/female). Further details of the dataset are listed in Table 1.

Training setup

Our methodology is designed to be versatile, i.e., compatible with various GAN architectures applicable to real-world datasets, as exemplified by architectures such as ProGAN (Karras et al., 2017), Large Scale GAN (Brock, Donahue & Simonyan, 2018), and StyleGAN (Karras, Laine & Aila, 2019). However, through experimentation with the Amazon fashion dataset we observed that the ProGAN architecture proposed by Karras et al. (2017) consistently outperformed others in generating high-quality images. Although we also trained the StyleGAN architecture for the same dataset, the results were not comparable to those achieved with ProGAN. This discrepancy can be attributed primarily to the considerable variance and diversity of images within each class present in the Amazon fashion dataset. Note: All GAN architectures are trained using the default hyperparameters recommended by their respective authors.

For optimizing our base latent vector, shifting vector, and the shift predictor Θ_s(⋅), we use the Adam optimizer with a learning rate of 0.01. Since the pre-trained ProGAN architecture takes the input of 512 dimensions, thus the size of both our base latent vector z and shifting vector k is set to 512. Additionally, each pixel in the output image generated by the pre-trained ProGAN model lies within the [0, 1] range. Therefore, throughout our method, we maintain all image data within this [0, 1] range. Further, to determine the optimal value for the search radius ϵ, we start with 0.1 and increment it by 0.05. After each increment, we manually review the generated image results. We found that ϵ=0.4 yields the best results, as values above this tend to generate images that deviate from the original class (e.g., given a shoe image, the shifted image becomes a shirt). We also fixed batch size n to 8 (can have any integer value) for all our experiments to reduce the time quantum to 1–2 min. We retrain the DVBPR recommendation model proposed by Kang et al. (2017). To train, we use a single Nvidia RTX A6000. Finally, to compare our work, we use personalized and non-personalized variants of image generation methods proposed by Kang et al. (2017) as our baseline because it is the only work in the literature that has some architectural similarities to our work, i.e., directly generating an entire image given a user’s preference. Both our method and Kang et al. (2017) method work at an image resolution of (64 × 64) scaled in later stages to (128 × 128) using bilinear interpolation.

Evaluation

In this subsection, we discuss the various evaluation metrics used to evaluate the generated images. Through this evaluation, we want to focus on three main criteria which are: uniqueness, quality, and user preference.

Results

In this subsection, we first discuss the results of quantitative experiments done to verify the performance of our method when compared to DVBPR (Kang et al., 2017). We use the metrics described in ‘Evaluation’ to test the quality, diversity, and user preference of the generated images. To begin with, since Kang et al. (2017) is a more class-centric approach, i.e., generates images that belong to a single class domain and are not item-specific, so we similarly change our methodology to have a suitable comparison. We remove our item-specific portion from the framework defined in Fig. 2 and generate images around randomly sampled latent vectors using the ProGAN generator (Karras et al., 2017). After making both approaches comparable, we calculate the specified metrics, whose values are listed in Table 2.

Table 2 compares our method with two variants of DVBPR method (Kang et al., 2017). The first variant of the baseline is the personalized one, where the generator learns to optimize the user preference score, and the second one is the random one, where the main goal of the generator is to only optimize the quality of the generated images. As can be seen, our method obtains a mean preference score of 8.97, thus improving by 16.7% over the mean preference score of 7.68 obtained by the personalized variant of Kang et al. (2017). Similarly, our methodology which is generating personalized images obtains a mean diversity (opposite SSIM) score of 0.63, thus improving by 12.5% over the non-personalized variant of Kang et al. (2017) and by 18.8% over the personalized variant of Kang et al. (2017). Furthermore, our method outperforms (Kang et al., 2017) personalized method by 7.4%. Figure 4 compares the images generated for the above-explained experiment using our method and Kang et al. (2017) personalized method. Here, we see that, due to our proposed latent shifting: images generated using our method have a lot of variety in aesthetical aspects when compared to Kang et al. (2017).

Since our method can also work on item level, i.e., has the ability to generate various images with high user preference for a randomly sampled user, we also perform item level experiments. Table 3 depicts the results of the item-level experiments, we perform with our method, where we first sample a randomly chosen user along with one preferred and one unpreferred item image and then generate novel images with our method. As can be seen in Table 3 for unpreferred items our method can generate unique, realistic images that obtain a mean incremental preference score shift (base image preference - generated image preference score) of 7.31. This signifies that in an ideal situation, any seller can use our method to make modifications to their item leading to improved rank in the user preference list. Qualitatively, Fig. 5 depicts the images generated around a given object with high user preference, as can be seen, our base item image is really close to the original item image. Moreover, there is a lot of variety in the shifted item image set aesthetically. Interestingly we also see a few images that are not related to the actual ground truth image. We believe that is happening because there is no constraint except for the search radius limitation on shifted images. Thus, a potential area to explore in the future.

Additionally, we also perform a more real-life experiment to bring out the pragmatic capabilities of our approach. Firstly, we perform various tests on our approach against mutually exclusive datasets. For this, we divide the Amazon-fashion dataset (McAuley et al., 2015) into three major segments, i.e., shirts, footwear, and trousers. Table 4 provides the results in line with metrics discussed in ‘Evaluation’. As seen in Table 4 our approach provides a mean incremental preference score shift (base image preference–generated image preference score) of 7.3 across different subsets of data. Thus establishing that our approach is highly transferable.

Another major real-life scenario is where a seller has to modify its product such that the product has a high preference across a large number of users (not just one). To test our approach for such a scenario we randomly sample different subsets of users of size n = {1, 5, 10, 50, 100, 500, 1000, 10000}. Table 5 depicts the results of this experiment, Here one can see that our approach for user counts greater than 10 (Fig. 6) provides a constant mean incremental preference score shift (base image preference–generated image preference score) of 3.2 without affecting the quality of the image depicted by the FID, Inception Score metric.