Development of a travel recommendation algorithm based on multi-modal and multi-vector data mining

Overall design

Figure 1 depicts the framework diagram of the multi-modal sentiment-aware tourism recommendation algorithm, which analyses text and image information on the tourism platform under the model framework diagram. Multi-modal topic mining and sentiment analysis are conducted using data from the tourism domain. After inputting a multi-modal corpus of tourists, the algorithm separately outputs multi-modal topics, including each tourist/attraction and their corresponding sentiment information.

Figure 2 illustrates the flowchart of the mining and analysis of text data in multi-modal data. The text is first parsed, and then the multi-vector word meanings are segmented according to the Chinese sub-word. Text feature vectors are subsequently constructed to complete text retrieval, clustering, classification, and association analysis.

Feature selection

Feature selection is a critical technique that directly impacts classification accuracy. Its primary purposes are to improve classifier efficiency by reducing the number of adequate words and to enhance classification accuracy by eliminating noisy features. As such, feature selection is one of the most effective technical methods for reducing feature dimensionality. By selecting relevant information features from a high-dimensional space, the learning process is accelerated, pattern generalization is improved, the running time of fundamental applications is reduced, and it plays a crucial role in many subject area applications. The feature selection process in this article is depicted in Fig. 3, where the construction of candidate feature subsets and evaluation are performed based on the original feature data set, the performance of the subset that satisfies the conditions is selected, and finally, the validation is concluded. Feature selection enables more effective and accurate data classification by eliminating irrelevant data during data dimensionality reduction, improving learning accuracy, and improving result comprehensibility.

Multi-modal data mining for tourist users

Multi-modal data of visitors refers to the combination of textual and visual data, where each visitor document contains text and image data. Textual data includes words and sentiment words generated based on the corresponding topic’s document-topic distribution and document-sentiment word distribution. Visual data includes visual words generated based on the document-topic distribution. The multi-modal data of visitors is used for topic modelling and sentiment analysis to identify each visitor’s preferences.

(1)${\displaystyle W_t\in 1,2,4,\dots ,w^u}$ (2)${\displaystyle V_t\in 1,2,4,\dots ,w^v}$ where W_t isthe text theme and V_t is the visual theme. Under the definition of each theme W_t, the polynomial distribution on the text theme and visual theme can be obtained according to ${\beta }_1^u$ and ${\beta }_2^u$.

(3)${\displaystyle {\phi }^u\in Dir\left({\beta }_1^u\right)}$ (4)${\displaystyle {\psi }^u\in Dir\left({\beta }_2^u\right)}$

For each visitor topic W_t ∈ 1, 2, 4, …, w^u, obtain the polynomial distribution over the sentiment words according to the hyperparameters of the Dirichlet distribution η^u. (5)${\displaystyle {\pi }^u\in Dir\left({\eta }^u\right)}$

At this point, for each document in the travel platform d^u , a polynomial distribution can be obtained for each document based on the Dirichlet parameter a^u. (6)${\displaystyle {\psi }_a^u\in Dir\left(a^u\right)}$

For each text word in the document d^u w, the subject $z_a^w\sim Multi\left({\psi }_a^u\right)$ is obtained according to the distribution ${\psi }_a^u$; then, get the text word $w\sim Multi\left({\psi }_{z_a^w}^u\right)$ based on the distribution ${\psi }_{z_a^w}^u$.

For each visual word in the document d^u v, first get the subject $z_a^v\sim Multi\left({\psi }_a^u\right)$ according to the distribution ${\psi }_a^u$, then get the text word $v\sim Multi\left({\psi }_{z_a^v}^u\right)$ based on the distribution ${\psi }_{z_a^v}^u$.

For each sentiment word in the document d^u s, first obtain the topic distribution $\tau \sim Uniform\left(z_1^u,z_2^u,z_3^u,\dots ,z_k^u\right)$. Then get the sentiment word s ∼ Multi(π^u) based on the distribution ${\pi }_1^u$. (7)${\displaystyle \left\{\begin{array}{c}w\sim Multi\left({\psi }_{z_a^w}^u\right)\hfill \\ v\sim Multi\left({\psi }_{z_a^v}^u\right)\hfill \\ s\sim Multi\left({\pi }^u\right)\hfill \end{array}\right.}$

So far, the three-word senses of Eq. (7) have been obtained. The feature expressions of the tags are constructed by decomposing each tag into spatial vector models. The distribution of conceptual features in images shared by users in different interest categories is different as are the posted travel texts and the user labels, so their interests can be predicted accordingly. Given a certain user U, suppose the set of images contained in his posted information is S = i₁, i₂, …, i_n, n denotes the number of images, and for each image i, the VGG19 model is used to extract image features. The process of partial image feature representation is shown in Fig. 4, and the output of the second fully connected layer (FC2) is selected as the feature vector of the image V_i . The fused multiple image feature vectors are averaged and input to the fully connected layer, and the output is used as the feature representation of the image part.

Finally, based on the text part feature V_t and the image part feature V_pic, the relationship matrix after multi-modal data fusion is obtained as follows: (8)${\displaystyle A=V_t\otimes V_{pic}}$ where, ⊗ represents the direct product operation. The mean value of matrix A is calculated by column, and the calculation result is input into the 2-layer fully connected layer, and the final judgment result is output.

Data processing

This section runs simulation tests to compare the performance of the proposed model in this study with the methods described in the literature to assess the effectiveness of the model (Javed et al., 2021; Park & Kim, 2021, and Fedorov et al., 2022) for comparative analysis. Firstly, the dataset is processed by crawling data from the travel website TripAdvisor, which is then used to construct the dataset for evaluating the comparative experiments. The multi-modal data, consisting of text and travel images, yields 17,398 tourists’ data, including 16,782 text comments and 35,628 images. In order to analyze the text data, text words are extracted. Since nouns are more likely to represent the objective meaning of things and are thus more suitable for describing the properties of things, all nouns in the documents are extracted as text words in this article. Additionally, adjectives, verbs, and adverbs are more likely to carry evaluation or emotional tendencies of things, and are thus used as emotional words for word sense segmentation. To distinguish the word markers in the document into nouns, adjectives, verbs, and adverbs, the lexical annotation function provided by Stanford NLP toolkits is utilized. In terms of image processing, the final visual content of the images is represented by 1,021 visual words. Specifically, the images are first chunked and clustered using the K-means method based on feature extraction, resulting in a final image.

Comparative performance

The efficacy of the proposed model in the travel recommendation task is evaluated based on a text feature length of 1,000 and image features of 1,000 to verify whether the multi-modal data fusion and mining method proposed in this article can improve the accuracy of interest classification to a significant extent. Moreover, it is verified that the content classification model structure based on image and text fusion is better than the traditional unimodal classification results. The number of fully connected layer nodes in Park & Kim (2021) is 4,096, with two fully connected layers, and each training input batch contains 64 training data. The dataset is divided into two parts for quantitative experiments, with 80% set as the training set and the remainder as the test set. Figure 4 displays the variation of each scenario’s perplexity scores under different topics. Based on the experimental results, 100 is chosen as the number of iterations for the experiment. It can be observed from the results that Allan et al. (2003) achieved the highest perplexity, which implies the worst generalization ability, as Allan et al. (2003) only models text words and sentiment words without distinguishing between them. Park & Kim (2021) and Fedorov et al. (2022) improve the generalization ability based on the additional dependency between visual or sentiment information and textual information. However, compared to the models in Allan et al. (2003), Park & Kim (2021), and Fedorov et al. (2022), the proposed model in this article performs better and obtains the best results.

Meanwhile, in order to evaluate the performance of the schemes in terms of F1 and Recall, relevant experiments were compared. Table 1 shows that the F1 and Recall values of the other schemes are all lower than 85%, and the model constructed by the text has the best effect, achieving the best results among all the models, with the F1 value of 86.4% and the recall rate of 86.1%. All indexes are higher than those of other models, indicating that our model is more suitable for the travel recommendation system, and its recall rate and F1 value will be increased to 3%–5%.

To assess the efficacy of the recommendation system, we selected 1,261 tourists who had visited at least 15 attractions after the completion of the model learning. In this study, we measured and evaluated the performance of the proposed algorithm using Precision and MAP following traditional search metrics. The Precision and MAP values for the two experiments, when LOPs are 5, 10, and 20, are presented in Figs. 5 and 6, respectively. Allan et al. (2003) performs poorly because it cannot explore the potential relationships between multi-modal topics and emotions. Park & Kim (2021) and  Fedorov et al. (2022) serve better than Allan et al. (2003) because they can capture the consistency between different modalities. It indicates that visual or emotional information can enhance recommendation performance. Our proposed model outperforms all experimental comparison models, demonstrating that an effective combination of textual data, visual data, and sentiment can enhance data mining functionality and assist in implementing recommendation applications.