Relation semantic fusion in subgraph for inductive link prediction in knowledge graphs

Path and rule-based link prediction

These approaches use path information or logical rules between entities in a knowledge graph to predict links. The Path Ranking Algorithm (PRA) (Lao & Cohen, 2010) analyses the path patterns between entities. AMIE (Galárraga et al., 2013), RuleN (Meilicke et al., 2018), Neural-Lp (Wang et al., 2019) and DRUM (Yang, Yang & Cohen, 2017) learn logical rules from the knowledge graph. AMIE and RuleN generate logical rules based on path search traversal, which has limited scalability. Neural-Lp generates logical rules through recurrent neural networks, while DRUM learns rules in a differentiable manner. These rules can capture semantic information that is independent of entities. However, they still face challenges in handling complex logic and large scale data, as well as improving the accuracy and interpretability of rules.

Embedding-based link prediction

Embedding-based link prediction predicts links by mapping entities and relations in the knowledge graph to embeddings in a low-dimensional space. The embeddings capture semantic information about the entities and relations, enabling the computation of possible relations between entities. TransE (Bordes et al., 2013) is one of the earliest models for embedding knowledge graphs, which assumes that the sum of the embeddings of the head and tail entities should be close to the relation embeddings. TransE is a model suitable for 1-to-1 relations. However, it has limitations when dealing with 1-to-many, many-to-1, and many-to-many relations. To handle complex relation types better, TransH (Wang et al., 2014) TransR (Lin et al., 2015), and TransD (Ji et al., 2015) are proposed. DistMult (Yang et al., 2015) and ComplEx (Trouillon et al., 2016) capture complex interactions between entities by using different scoring functions. ComplEx introduces complex embeddings to model symmetric and antisymmetric relations. With the development of neural network technology, neural network-based knowledge graph representation models have shown good performance in representation learning, M-DCN (Zhang et al., 2020) utilize multi-scale processing to optimize quality in the convolution layer for knowledge graph embedding. It generates various sizes of convolution filters to learn different characteristics between the embeddings of entity and relation. SHGNet (Li et al., 2023) designs a hierarchical aggregation architecture for feature propagation. And the method incorporates neighbor information via node aggregation and relation weighting, which can selectively aggregate informative features. TEGS (Li et al., 2024) amalgamates textual encoding and graph structure information, facilitating the concurrent acquisition of contextualized and structured knowledge. These approaches assume that all entities and relations are known during training. They cannot predict entities and relations that have not appeared in the training set. For knowledge graphs where new nodes and edges are constantly being added, embedding-based link prediction methods require frequent re-training to adapt to the changes.

GNN-based link prediction

In recent years, graph neural networks (GNNs) (Scarselli et al., 2008) have made significant progress in link prediction tasks. GNNs efficiently capture both local and global information in the graph structure. Graph Convolutional Network (GCN) (Kipf & Welling, 2016) learns the representation of nodes by performing convolutional operations on graphs. Graph attention network (GAT) (Velickovic et al., 2017) introduces an attention mechanism that allows the model to dynamically focus on important neighboring nodes. The relational graph convolutional network (R-GCN) (Schlichtkrull et al., 2018) is specifically designed to deal with heterogeneous relations in graphs. These methods enhance the expressive power of the model and improve the accuracy of link prediction, however, they cannot predict relations between newly added entities.

To achieve efficient generalization of patterns and predict relations between new entities without frequent model retraining, innovative approaches have been developed. These approaches combine subgraph extraction techniques with GNNs for inductive link prediction. GraiL (Teru, Denis & Hamilton, 2020) extracts an enclosing subgraph of target triplet from knowledge graph, and models the subgraph with GNN to capture the topological structure. CoMPILE (Mai et al., 2021) highlights the edge direction in the enclosing subgraph based on GraiL. TACT (Chen et al., 2021) considers the semantic correlation between two relations in the knowledge graph based on subgraph structural information. SNRI (Xu et al., 2022) contains the neighborhood path information of all nodes in the enclosing subgraph. RMPI (Geng et al., 2023) first transforms a triplet’s surrounding subgraph in the original KG into a new relation view graph, where inter-relation features are more straightforwardly represented. It then learns the embedding of an unseen relation from the relational subgraph by the relational message passing network. LCILP (Mohamed et al., 2023) extracts subgraph using a PPR-based local clustering technique.

Although these methods have achieved good results, they only use node information in the graph when selecting nodes for subgraph extraction, without considering the influence of edges, which are important components of knowledge graph.

Subgraph extraction

Most of the current inductive link prediction methods rely on the subgraph around the target relation, where the nodes of the subgraph are selected based on their proximity to the target nodes. Inspired by GENI (Park et al., 2019), we introduce a novel subgraph extraction method that constructs the subgraph by scoring nodes based on a combination of node distances and relational semantics. The subgraph extraction method is composed of four parts: (1) Calculation of the semantic importance score of nodes using attention mechanisms. (2) Computation of the structural importance scores of nodes using the PPR algorithm. (3) Calculating the final importance score of nodes by weighting and summing their structural and semantic importance scores. λ is a weight paremeter that adjusts the proportion of structural and semantic information in the calculation of node importance. (4) Selecting nodes to form a subgraph based on their scores. Specifically, nodes associated with the target relation are initially ranked in descending order based on their importance scores. Subsequently, these nodes are incrementally added to an initially empty set of seed nodes. After each node addition, the conductance of the current set of seed nodes is computed. The process stops when the calculated conductance value falls below a predefined threshold. The resulting seed node set and its corresponding local subgraph is the subgraph we need.

 
_______________________ 
Algorithm 1 Subgraph extracting algorithm.___________________________________________ 
Require: graph G = (V, E) with vertices V and edges E; seed set S= {m, n}; 
     teleportation probability α ∈ (0, 1]; residual error β; weight paramater λ; 
     the embedding of nodes h; conductance threshold ϕ; 
Ensure: subgraph S 
  1:  for v in G do: 
  2:       scorestru =APPR(G,v, α, β); 
  3:       scoreseman =semantic_weight(G,h); 
  4:       score = λ× scorestru +(1-λ)× scoreseman; 
  5:  end for 
 6:  sort nodes in descending order according to their scores 
  7:  compute the conductance of the set S; 
  8:  while conductance < ϕ do: 
  9:       add node to the set of seed nodes S; 
10:  end while 
11:  return S;_____________________________________________________________________________________    

1. Semantic importance score for node

In knowledge graphs, relations contain crucial semantic information essential for understanding the graph structure and conducting effective node importance assessment. Different semantic relations exert varying influences on node importance. For the target relation ‘lives in’ in Fig. 4, the relations ‘parent of’, ‘work in’, and ’located in’ are less important than ‘be good at’, ‘is friend of’, and ‘husband of’. Nodes connected to more relevant relations should receive higher priority in importance assessment.

GATs are effective in capturing the relative importance between nodes, so we adopt the attentation weight as the semantic score of the node. However, GAT may overlook the relational semantic information when dealing with knowledge graphs. To address this issue, we integrates relational information into the computation of attention weights by sharing relation embeddings, as shown in Fig. 5. At each layer of the model, node features are passed to the next layer through the output of the previous layer, while edge features remain shared across all layers. The node updates its features by considering not only its information but also the relational information around it. This approach allows for a more accurate assessment of the importance of nodes in the knowledge graph In graph attention calculation, the weight coefficient e_ij from node v_i to node v_j is defined as (1)${\displaystyle e_{ij}=a\left[s\left(i\right)\parallel \sum _{m=1}^k{p}_{ij}^m\parallel s\left(j\right)\right]}$ where s(i) is the embedding of node i obtained by other ways such as TransE or one-hot. Since there can be many different types of relations between two entities in the knowledge graph, in order to distinguish different types of edges, k is the number of edge between node i and j, $p_{ij}^m$ represents the embedding of the m-th edge. a is an attention calculation function. All the computed weight coefficients are normalized by the softmax function: (2)${\displaystyle {\alpha }_{ij}=\frac{exp\left(LeakyRelu\left(e_{ij}\right)\right)}{\sum _{k\in N_i}exp\left(LeakyRelu\left(e_{ik}\right)\right)},e_i\in N_i}$ where LeakyRelu is the activation function. N_i is the set of nodes in the graph.

2. Structural importance score based on the PPR algorithm.

The PPR algorithm is primarily used to determine the structural importance score of entities in the knowledge graph. It sets the head and tail entities of the target relation as the seed node set, then performs a random walk with a restart from the seed node set, it returns to the start node with probability c and jumps to neighbor nodes with probability 1-c. After many iterations, the algorithm reaches a stable value. The probability distribution can be viewed as the structural score score_stru link to the other nodes of the seed node: (3)${\displaystyle \mathbf{r}=cW+\left(1-c\right)\mathbf{e}}$ where c ∈ (0, 1], W denotes the transfer probability matrix, W[i,j] represents the probability from node j to node i. e denotes the initialization vector, If the start node is i, then e [i] equals 1. Otherwise, e [i] equals 0. The vector r is a column vector that represents the probability of being at node i. To reduce computational complexity, an approximate PPR algorithm is commonly used.

3. Node importance score

The final importance score of nodes is obtained by weighting and summing the structural importance score and semantic importance score. The score is defined as follows: (4)${\displaystyle score=\lambda \times scor{e}_{stru}+\left(1-\lambda \right)\times scor{e}_{seman}}$ where λ denotes the weight given to the structural importance score, and (1 −λ) denotes the weight given to the semantic importance score.

The entities should be sorted in descending order based on their score and added to the seed set S. Then, the conductance of the set S should be calculated. If the conductance reaches the specified threshold, the set can be used as an extracted subgraph of the target relation. The calculation of the conductance is defined as: (5)${\displaystyle \varphi \left(S\right)=\frac{\left|\left\{\left(i,j\right)\in E,i\in S,j⁄\in S\right\}\right|}{min\left(vol\left(S\right),2m-vol\left(S\right)\right)}}$ where E is the set of edges, i is a point in set S, j is a point outside of set S, and vol(S) is the sum of the degrees of all points in S, m is the number of edges in graph, 2m-vol(S) denotes the sum of the degrees of the nodes not included in S.

Initialization representation of nodes

An appropriate initialization can deploy the intrinsic properties and structural information of entities, and it can bolster the model’s representational capacity and generalizability. Most inductive link prediction models for subgraph node initialization adopt structure-based or attribute-based initialization. However, these methods fail to take into account the impact of relations in initializing the embeddings.

In a triplet, entities with the same relation are often semantically similar. Consequently, the relational is instrumental in capturing the essence of both the head and tail entities within the representational framework. By incorporating relation information associated with the entity into the entities themselves, it can implicitly indicate the specific category of the entity at the semantic level, thereby providing more useful information for the representation of new entities. Based on the aforementioned concept, when initializing the node embedding, the relations associated with the entity are aggregated to form the semantic information embedding of the entity’s relation. For entity e, there are both relations to and from it. To clarify the direction of the relation, we use r_t to represent the relation embedding pointing to the entity, and r_h to represent the relation embedding apart from the entity. Since an entity may have multiple relations with other entities, the semantic embedding of entity e is formed by aggregating all the relation embeddings around it, it is defined as (6)${\displaystyle e_r=\frac{1}{N_r}\left(\sum _{i=0}^{n_1}{r_h}^i+\sum _{j=0}^{n_2}{r_t}^j\right)n_1+n_2=N_r}$ where n₁ denotes the number of edges apart from entity e, n₂ denotes the number of edges pointing to entity e, and N _r denotes the total number of edges connected to entity e. r_hⁱ denotes the relation embedding originating from entity e, and r_tⁱ denotes the relation embedding directed from entity e.

The representation of entity e is composed of structural information embedding and semantic information embedding e_r. (7)${\displaystyle e=e_s\parallel e_r}$ the structural information embedding can be obtained by DRNL (Zhang & Chen, 2018) in GraiL.

Subgraph representation

The subgraph representation module uses RGCN to learn the representations of entities in subgraphs, and it is defined as: (8)${\displaystyle h_i^{\left(k+1\right)}=\sigma \left(\sum _{r\in R}\sum _{v_j\in N{v}_i^{\left(r\right)}}\frac{1}{a_{i,r}}{W}_r^{\left(k\right)}{h}_j^{\left(k\right)}+W_o^{\left(k\right)}{h}_i^{\left(k\right)}\right)}$ where R denotes the set of all relations in the extracted subgraph, and $N{v}_i^{\left(r\right)}$ denotes the set of neighbors that have relation r with node v_i.a_i,r is used to perform the normalization operation. W_r is the transformation matrix over relation r in the k-th layer. w_o is the weight parameter corresponding to the node itself, σ() is the activation function, and $h_i^{\left(k\right)}$ represents the embedding of node h_i in the k-th layer. The pooled average of all nodes in the subgraph is used as the subgraph representation h_s: (9)${\displaystyle h_{{}_{s_{}}}^K=\frac{1}{\left|{V_s}_{{}_{}}\right|}\sum _{i\in {V_s}_{{}_{}}}{h}_i^K}$ where V_s is the set of node, h_i denotes the embedding of node i.

Scoring and loss function

To obtain the likelihood score of the triplets, the scoring function is designed as follows: (10)${\displaystyle f\left(h,r,t\right)=\left[E_h^K\oplus E_t^K\oplus E_r\oplus E_g^K\right]W}$ where $E_h^K,E_t^K$, $E_g^K$ are the embeddings of the head entity, tail entity and subgraph in the K-th layer, E_r is the representation of the relation, and W is the weight matrix, and the loss is computed by summing up the scores of all triplets in knowledge graph: (11)${\displaystyle Loss=\sum _{\left(h_i,r_k,t_i\right)\in G}^{\left|N\right|}max\left(0,f\left(h_i,r_k,t_i\right)-f\left(\overline{h_i},r_k,\overline{t_i}\right)+\gamma \right)}$ where $\left|\mathrm{N}\right|$ denotes the number of triplets in knowledge graph, G denotes the knowledge graph, γ is margin hyperparameter. f(h_i, r_k, t_i) is the score of positive triplets, and $f\left({\bar{h}}_i,r_k,{\bar{t}}_i\right)$ is the score of negative triplets. In this paper, the positive triplets are the triplets existing in the knowledge graph, and the negative triplets are constructed by replacing the head (or tail) of the triplets with uniformly sampled random entities.

DataSet

FB15K is a subset of the FreeBase Knowledge Base (KB) (Bollacker et al., 2008), which contains general facts about the world with various relation types. FB15K237 is a subset of FB15K, created by removing reversible relational data. The dataset contains 15k subject terms and a total of 237 relations. WN18 is a dataset derived from WordNet KB (Miller, 1995), which is a comprehensive database of English vocabulary used to capture lexical relations, such as hyper-subordination between words. WN18 contains a significant number of reversible relations, but WN18RR removes these reversible relations to improve the accuracy of the link prediction task. WN18RR only encompasses 11 relations. To evaluate the model’s link prediction capability on a new graph, we adopts GraiL’s dataset partitioning method. Four entities are randomly selected as root nodes from the dataset, and the k-hop neighborhood nodes around each root node are merged to create the training dataset. The sample training graph is then removed from the entire graph, and the same method is applied to the remaining graph to obtain the test graph. The process parameters are adjusted to generate a series of graphs with increasing size. The dataset is divided into four parts, denoted as v1, v2, v3, and v4, based on the increasing number. The same method is used to obtain the training and test sets for WN18RR. The dataset statistics are presented in Table 1.

Results and discussion

To compare the performance of SASILP with existing inductive link prediction models, we conducted link prediction experiments on the benchmark datasets WN18RR and FB15K237. We referenced the publicly reported experimental results of the baseline models Neural-LP, DRUM, RuleN and GraiL. However, due to incomplete information in the literature, the results for the SNRI, LCILP and RMPI models are obtained through local execution on our machine.

• Results

Several observations can be obtained from Tables 2 and 3

(1) SASILP outperforms the rule_based models including Neural-LP, DRUM, and RuleN in all metrics. Compared to the best-performing RuleN model, our method has achieved significant improvements across various metrics; specifically, on dataset WN18RR_v3, MRR has been enhanced by 7.27%, and Hits@10 has been improved by 13.88%. Similarly, on FB15K237_v2 dataset, our approach has demonstrated outstanding performance, with MRR showing an improvement of 1.94% and Hits@10 by 4.72%. This is likely due to SASILP model is not only learn path-based rules, but exploit complex patterns in knowledge graph.

(2) Our model performs better in most cases than the latest subgraph_based model including Grail, SNRI, LCILP and RMPI. Compared with the best-performing model LCILP, our method has also shown improvements, specifically, on dataset WN18RR_v3 MRR has been enhanced by 0.77%, and Hits@5 has been improved by 2.4%. It indicates that incorporating relational semantics into subgraph extraction has a beneficial influence on the inductive link prediction experiments.

(3) On FB15K237, RuleN outperforms GraiL on all datasets for Hits@1. This is likely due to RuleN focus on the existence of the rule-based path (i.e., 1 or 0) rather than calculating the probability of the path. However, for Hits@10, GraiL outperforms RuleN, suggesting that GraiL performs better for a wider range of link predictions.

The performance improvements substantiate the effectiveness of our model in the task of inductive link prediction.

• Performance comparison of different subgraph extraction strategies

We conduct additional experiments on dataset WN18RR to analyze the impact of hop selection in extracting subgraphs. The values of MRR, Hits@1, Hits@5, and Hits @@10 were taken for k = 1, 3 and 5, respectively. The results are shown in Fig. 6.

Figure 6 shows that the performance in terms of MRR, Hits@1, Hits@5, and Hits@10 is the least favourable on all datasets when k = 1. However, there is not much difference when k = 3 and k = 5. This may be due to the fact that the subgraph consisting of 1-hop neighbors includes a small number of nodes, which limits the availability of contextual semantic information for relation prediction. When k = 3, subgraphs can provide sufficient contextual semantic information to improve link prediction performance. However, the number of nodes in the subgraph increases when k equals 5, and the performance of inductive link prediction does not improve significantly. On the contrary, it decreases on some datasets. This suggests that although an increase in the number of hops introduces more information about nodes, not all of the introduced nodes are necessarily useful for link prediction. Some of them may be noisy nodes, which do not benefit the performance of relation prediction.

• Ablation study

We conduct ablation studies on FB15K237_v2 to validate the importance of relation semantics for subgraph extraction in SASILP. Table 4 shows the results of the ablation study. After removing the relation semantic score in subgraph extraction, the MRR value reduces by 8.34%, and the Hits@5 value reduces by 18.36%. This result demonstrates the effectiveness of relation semantics in inductive link prediction.

• Case Study

To analyze the subgraph under different subgraph extraction strategies, we selected two nodes (07s9rl0, 06 × 77 g) in FB15K237_v2 as seed nodes. The subgraphs extracted using GraiL contain 1,218 nodes, the value of Hits@10 is 52.2%, While SASILP extracted using SASILP contains 11 nodes, the value of Hits@10 is 60.02%. Figure 7 shows the semantics of the extracting subgraph, where the purple circle represents the two seed nodes and the yellow dotted line represents the relation to be predicted. The number of subgraph nodes decreased sequentially, but it did not influence the link prediction performance. It may be caused that the nodes in the subgraph have a close semantic correlation with the missing relation, and it also demonstrates the effectiveness of our subgraph extraction method.