Effective attributed network embedding with information behavior extraction

SNE methods

DeepWalk (Perozzi, Al-Rfou & Skiena, 2014) employs Skip-Gram (Mikolov et al., 2013) to learn the representations of nodes in the network. It uses a random selection of nodes and truncated random walk to generate random walk sequences of fixed length. Subsequently, these sequences are transported to the Skip-Gram model to learn the distributed node representations. LINE (Tang et al., 2015) studies the problem of embedding very large information networks into low-dimensional vector spaces. Node2vec (Grover & Leskovec, 2016) improves the strategy of random walk and achieves a balance between BFS and DFS. SDNE (Wang, Cui & Zhu, 2016) proposes a semi-supervised deep model, which can learn a highly nonlinear network structure. It combines the advantages of first-order and second-order estimation to represent the global and local structure attributes of the network. Besides, there are many other SNE methods (Goyal & Ferrara, 2017), which systematic analysis of various structural graph embedding models, and explain their differences. Nevertheless, these methods fully utilize structural information but do not consider attribute information.

ANE methods

CANE (Tu et al., 2017) proposed an approach of network embedding considering both node text information of context-free and context-aware. CENE (context-enhanced network embedding) (Sun et al., 2016) regards text content as a special kind of nodes and leverages both structural and textural information to learn network embeddings. TopicVec (Li et al., 2016) proposes to combine the word embedding pattern and document topic model. JMTS (Alam, Ryu & Lee, 2016) proposes a domain-independent topic sentiment model to integrate topic semantic information into embedding. ASNE (Liao et al., 2018) adopts a deep neural network framework to model the complex interrelations between structural information and attributes. It learns node representations from social network data by leveraging both structural and attribute information. ABRW (Hou, He & Tang, 2018) reconstructs a unified denser network by fusing structural information and attributes for information enhancement. It employs weighted random walks based network embedding method for learning node embedding and addresses the challenges of embedding incomplete attributed networks. AM-GCN (Wang et al., 2020) is able to fuse topological structures, and node features adaptively. Moreover, there exist quite a few survey papers (Jiao et al., 2021; Zhang et al., 2020; Daokun et al., 2017; Peng et al., 2017; Cui et al., 2019), which provide a comprehensive, up-to-date review of the state-of-the-art network representation learning techniques. They cover not only early work on preserving network structure, but also a new surge of incorporating node content and node labels.

It is challenging to get network embedding considering attributes of local context and topic due to its complexity. Quite a few works have been carried out on this issue. However none of them consider node information behavior features in attributed networks.

Information behavior extraction framework (IBE)

The information behavior features generated based on topics are complementary to features of existing models. As shown in Fig. 2, data sources of the framework IBE consist of two parts. One is the network embedding Z^B generated by one of existing embedding methods, and the other is Z^I, where $Z^B\in {\mathbb{R}}^{|V|\times d^{\mathrm{\prime }}}$ and $Z^I\in {\mathbb{R}}^{|V|\times d}$ (d′, d ≪ |V|) are embedding matrix consisting of the embedding vectors $\overleftarrow{z_v^B}$ and $\overleftarrow{z_v^I}$ of nodes V, respectively. We linearly concatenate the embedding matrix Z^B and Z^I to generate a joint embedding matrix Z, which can be used for link prediction, recommendation, and other tasks in attributed networks.

Topic-sensitive network embedding model (TNE)

In this section, we present the process of extracting node information behavior features. As shown in Fig. 3, the TNE model consists of two parts: an ISR algorithm (Importance score rating algorithm) and a topic-sensitive embedding matrix (Γ) transposing.

Generating node joint embedding vectors based on IBE

Z^B is a basic embedding matrix trained by one of existing embedding methods. Each row of the Z^B is a basic vector $\overrightarrow{z_v^B}$ for a node v generated by one of existing embedding methods.

Before getting Z by Eq. (7) according to the framework IBE, we firstly enlarge Z^I or Z^B by λ (Eq. (5)) so that the element values of λ * Z^I and Z^B or Z^I and ${\displaystyle \frac{Z^B}{\lambda }}$ are of the same order of magnitude, and the λ is calculated as follows.

(5) $$\lambda ={\displaystyle \frac{\overline{|b|}}{\overline{x}}=({\displaystyle \frac{\sum _{i=0}^{|V|-1}\sum _{j=0}^{d-1}|b_{ij}|}{|V|\ast d})÷({\displaystyle \frac{\sum _{i=0}^{|V|-1}\sum _{j=0}^{\tau -1}{x}_{ij}}{|V|\ast \tau })}}}$$where $\overline{|b|}$ is the average of all elements in Z^B, and $\overline{x}$ is the average of all elements in Z^I. And then we enlarge the element values of Z^I or Z^B who has the larger AUC (Hanley & Mcneil, 1982) value by weight coefficient α (Eq. (6)) again.

(6) $$\alpha =[auc(Z^I)÷auc(Z^B){]}^{\psi }$$where auc() is a function used to calculate the value of AUC, ψ is an amplification factor of the ratio ${\displaystyle \frac{auc(Z^I)}{auc(Z^B)}}$.

Especially, we should not use the method of reducing the element values of Z^I or Z^B to make their element values of the same order of magnitude, because it may result in invalid results due to the element values are too small. So, according to the values of coefficients α and λ, we divide the methods of linearly concatenating Z^I and Z^B into four cases as follows.

(7) $$Z=(\overrightarrow{z_{v_0}},\overrightarrow{z_{v_1}},\dots ,\overrightarrow{z_{v_{(|V|-1)}}}{)}^T=\{\begin{array}{ccc}& [(\alpha \ast \lambda \ast Z^I)\parallel Z^B],& \mathrm{i}\mathrm{f}\lambda \ge 1\mathrm{a}\mathrm{n}\mathrm{d}\alpha \ge 1\\ & [(\lambda \ast Z^I)\parallel {\displaystyle \frac{Z^B}{\alpha }],}& \mathrm{i}\mathrm{f}\lambda \ge 1\mathrm{a}\mathrm{n}\mathrm{d}\alpha <1\\ & [(\alpha \ast Z^I)\parallel {\displaystyle \frac{Z^B}{\lambda }],}& \mathrm{i}\mathrm{f}\lambda <1\mathrm{a}\mathrm{n}\mathrm{d}\alpha \ge 1\\ & [(Z^I)\parallel {\displaystyle \frac{Z^B}{\alpha \ast \lambda }],}& \mathrm{i}\mathrm{f}\lambda <1\mathrm{a}\mathrm{n}\mathrm{d}\alpha <1\end{array}$$where the operator [·||·] denotes concatenation, α is an enlarging coefficient to make the joint embedding matrix Z more similar to Z^I or Z^B who has the higher AUC value, and λ (Eq. (5)) denotes the enlargement factor who try to be adjusted to make the element values of λ * Z^I and Z^B or Z^I and ${\displaystyle \frac{Z^B}{\lambda }}$ in the same order of magnitude.

For the case of [(α * λ * Z^I) ‖ Z^B] (λ ≥ 1 and α ≥ 1) in Eq. (7), Z is displayed in matrix form as follows:

$$Z=\left[\begin{array}{c}\alpha \ast \lambda \ast x_{00}\cdots \alpha \ast \lambda \ast x_{0(\tau -1)}{b}_{00}\cdots b_{0(d-1)}\\ \alpha \ast \lambda \ast x_{10}\cdots \alpha \ast \lambda \ast x_{1(\tau -1)}{b}_{10}\cdots b_{1(d-1)}\\ ⋮\ddots ⋮⋮\ddots ⋮\\ \alpha \ast \lambda \ast x_{(|V|-1)0}\cdots \alpha \ast \lambda \ast x_{(|V|-1)(\tau -1)}{b}_{(|V|-1)0}\cdots b_{(|V|-1)(d-1)}\\ \end{array}\right],$$where each row of Z is the final joint embedding vector $\overrightarrow{z_v}$ for node v based on the framework of IBE and x_ij (0 ≤ i < |V|, 0 ≤ j < τ) is the element of $\overleftarrow{z_{v_i}^I}$, b_ij (0 ≤ j < d) is the element of $\overrightarrow{z_{v_i}^B}$. The other three cases of Eq. (7) have similar matrix representations.

Datasets

In Table 1, we consider the following real-world network datasets. BlogCatalog (http://networkrepository.com/soc-BlogCatalog.php) is asocial blog directory. The dataset contains 39 topic labels, 10,312 users, and 667,966 links. Zhihu is the largest online Q & A website in China. Users follow each other and answer questions on this site. We randomly crawl 10,000 active users from Zhihu and take the descriptions of their concerned topics as text information (Tu et al., 2017). The topics of Zhihu are obtained by the fastText model (Joulin et al., 2016) and the ODP of predefined topic categories (Haveliwala, 2002). The fastText presents the hierarchical softmax based on the Huffman tree to improve the softmax classifier taking advantage of the fact that classification is unbalanced in CBOW (Joulin et al., 2016). WiKi contains 2,408 documents from 17 classes and 17,981 edges between them. Cora is a research paper classification citation network constructed by McCallum et al. (2000). After filtering out papers without text information, 2,277 machine learning papers are divided into seven categories and 36 subcategories in this network. Citeseer is divided into six communities: Agents, AI, DB, IR, ML, and HCI and 4,732 edges between them. Similar to Cora, it records the citing and cited information between papers.

Baselines

To validate the performance of our approach, we employ several state-of-the-art network embedding methods as baselines to compare with our IBE framework. A number of existing embedding methods are introduced as follows.

• CANE (Tu et al., 2017) learns context-aware embeddings with mutual attention mechanism for nodes, and the semantic relationship features are extracted between nodes. It jointly leverages network structure and textural information by regarding text content as a special kind of node.

• DeepWalk (Perozzi, Al-Rfou & Skiena, 2014) transforms a graph structure into a sample set of linear sequences consisting of nodes using uniform sampling. These linear sequences are transported to the Skip-Gram model to learn the distributed node embeddings.

• HOPE (Ou et al., 2016) is a graph embedding algorithm, which is scalable to preserve high-order proximities of large scale graphs and capable of capturing the asymmetric transitivity.

• LAP (Belkin & Niyogi, 2001) is a geometrically motivated algorithm for constructing a representation for data sampled from a low dimensional manifold embedded in a higher dimensional space.

• LINE (Tang et al., 2015) learns node embeddings in large-scale networks using first-order and second-order proximity between the nodes.

• Node2vec (Grover & Leskovec, 2016) has the same idea as DeepWalk using random walk sampling to get the combinational sequences of node context, and then the network embeddings of nodes are obtained by using the method of word2vec.

• GCN (Kipf & Welling, 2016) model uses an efficient layer-wise propagation rule based on a localized first-order approximation of spectral graph convolutions. The GCN model is capable of encoding graph structure and node features in a scalable approach for semi-supervised learning.

• GAT (Veličković et al., 2018) is a convolution-style neural network that operates on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of methods based on graph convolutions. GAT enables implicitly specifying different weights to different nodes within a neighborhood.

Evaluation metrics and parameter settings

We randomly divide all edges into two sets, training set and testing set, and take a standard evaluation metric AUC scores (area under the ROC curve) (Hanley & Mcneil, 1982) as evaluation metrics to measure the link prediction performance. AUC represents the probability that nodes in a random unobserved link are more similar than those in a random nonexistent link. Because the number of topics is different in each dataset, we use τ to denote the maximum number of topics for each dataset. In concatenating weight coefficient α = [auc(Z^I) ÷ auc(Z^B)]^ψ, we set the factor ψ equal to 4 except that ψ has a specified value.

Experimental results

For experiments, we evaluate the effectiveness and efficiency of our IBE on five networks for the link prediction task. For each dataset, we compare the AUC data of basic embedding matrix Z^B generating by one of the existing embedding methods, the information behavior feature vectors Z^I, and their joint embedding vectors Z generating by the framework IBE. We employ six state-of-the-art embedding methods as baselines, including Node2vec, DeepWalk, LINE, LAP, CANE, HOPE, for comparisons with their extending frameworks IBE in the following experiments.

Table 2 compares AUCs over five datasets. By concatenating Z^B and Z^I linearly, the joint embedding vectors Z achieves the best performance. Especially on the BlogCatalog and Zhihu datasets, the AUC values of the joint embedding vectors Z are higher, compared with their baselines, than 14.6% and 27.6% respectively on average. One of the reasons may be that the average AUC of information behavior feature vectors Z^I are 81.7 and 82.4 respectively, which are more than 11% higher compared with the other three datasets on average. The other reason is that the maximum topic number τ of BlogCatalog and Zhihu are larger than those of Cora and Citeseer. In Table 2, we can also see that most of the AUC values of the joint embedding vectors Z for datasets Wiki and Cora exceed 90%. The reason is that their AUC values of baselines are relatively high, most of them are more than 85%. On the Citeseer dataset of Table 2, we also can see that the improvement of the AUC values of the joint embedding vectors Z is less, compared with their baselines. The result can be explained by the fact that the AUC values of information behavior feature vectors Z^I are all low, about 55% and the direct reason for the low AUC values is that the number of topics is too small. Due to the number of topics is small, the topic subdivision degree of nodes is low, and the classification of node labels will not be too detailed.

In general, in Table 2, it can be seen that the AUC values of concatenating embedding vectors Z are higher than that of Z^B and Z^I, which indicates that the concatenating method can properly integrate the features of all parties.

Ablation experiments: To investigate the effectiveness of TNE, we perform several ablation studies. In Table 2, the Z^B is the basic embedding generated by an existing model. The Z^I is obtained by the TNE model. A joint embedding Z is obtained by adding a Z^I to a Z^B, shown in Fig. 2 and Eq. (2). We observe that the quality of the joint embedding Z is better than itself Z^B.

Parameter sensitivity analysis

We further performed parameter sensitivity analysis in this section, and the results are summarized in Figs. 4 and 5. Due to space limitations, we only take the dataset of Wiki, Zhihu as an examples to estimate the topic number j (0 ≤ j ≤ τ) and the amplification factor ψ of vector concatenation can affect the link prediction results.

The topic number j (0 ≤ j ≤ τ): In Fig. 4, we illustrate the relationship between the number of topics and link prediction, where the order of selection of node topics is random. When j = 0, Z^I does not exist, Z degenerates to Z^B. As shown in Fig. 4, we can see that as j increases from 1 to τ, Z^B linearly combines with more topic-based feature dimensionality from Z^I, and the AUC values keep changing. When the AUC values of Z^B are below 82% in Fig. 4A, AUC values increase sharply with an increase of j. When the AUC values of Z^B are higher than a certain critical value, the AUC values increase more slowly or even stop growing with an increase of j.

So, we can see that when the number of topics is large in a dataset, each node can be classified in detail by the topic classification labels, which helps to improve the AUC values using a small number of topics. These also show that the AUC values of the concatenating embedding vectors Zs will be higher than that of all parties for concatenating, that is Z^Is and Z^Bs, but it will not increase indefinitely.

The amplification factor ψ of vector concatenation: ψ is an amplification factor for α (Eq. (6)) which is a weight coefficient for enlarging the element values of Z^I or Z^B who has the larger AUC (Eq. (7)). From Fig. 5, we can see that the AUC value, when ψ is less than 0, is less than that when ψ is greater than 0. The reason is that the weight coefficient α, when ψ is less than 0, enlarges the Z^I or the Z^B who has the smaller AUC value. As a result, the joint embedding Z is more similar to one that has a lower AUC value. When ψ is between 1 and 5, the prediction result is the best. However, when the value of ψ increases gradually, the AUC values decrease slightly and tend to the Z^Is or the Z^Bs who have the larger AUC values.