Human-computer interaction based on background knowledge and emotion certainty

Problem definition

When two interacting parties engage in a conversation about a certain topic, as the conversation continues, the content related to the topic gradually increases and the background knowledge of both parties is gradually awakened. In the current HCI system, the participant is the one who has the background knowledge in the interaction process, and if the dialogue system does not introduce new content or introduces content that is not relevant enough or not emotionally desired by the participant, it will largely reduce the participant’s willingness to have a dialogue and thus lead to the end of the dialogue. In this study, we introduce a knowledge graph as the background knowledge of the interaction system to simulate human-to-human communication, so that the interaction system can take into account the emotional state of the participants and continuously provide emotionally and content-related dialogue content.

The human-computer interaction process is shown in Fig. 1, where H and R represent the participant and robot in the human-computer interaction, respectively. The content input by the participant to the interactive system at the kth time is $C_H^k$, the content of the robot’s reply to the participant is $C_R^k$. Let the friendliness of emotional interaction between participant and robot as R(k), if we need to get the kth conversation content $C_R^k$, the specific mathematical expression is

(1) $$f:\left(R\left(k\right),\mathit{G},C_H^k\right)\to C_R^k$$

Inputs and outputs:

The knowledge graph G is introduced at system initialization; during the interaction, the participant input is the kth conversation content $C_H^k$, and the system output is the kth robot’s response content $C_R^k$.

In the human-computer interaction process, the following two factors should be considered:

(1) Assessment of the human-computer interaction emotional relationship: human-computer interaction is a continuous interaction process, and the participants’ perceptual affection for content affects the continuity of dialogue. Based on this, the interactive friendliness of emotional perception in human-computer dialogue was evaluated first. In the process of interaction, the status of emotional perception of participants showed an overall upward trend, which was considered to be in a good interactive relationship.

(2) Assessment of the relationship between human-computer interaction content: the process of communication between people is a process of awakening background knowledge, and there is a correlation between the contents of communication. Based on this, the relevance assessment is carried out on the content of human-computer dialogue, and the potential interested content of participants is found on the knowledge graph.

Emotional friendliness

Human-computer interaction is a continuous process, and the current affective state is not only related to the content of the current interaction conversation, but also correlates with the content of the historical interaction sessions. The update function of the HCI affective friendliness R(k) is defined by combining the current and historical interaction sessions as

(2) $$R\left(k\right)=min\left(1,max\left(R\left(k-1\right)+W\left(k\right)\times C\left(k\right),0\right)\right)$$where R(k) is the kth human-computer interaction emotion friendliness and takes values in the range [0, 1], where a smaller value indicates a worse emotional interaction state and, conversely, a larger value indicates a better emotional interaction state. In particular, an initial value of 0.5 indicates an uncertain human-computer interaction. W(k) is the interaction input affective evaluation value, with an initial value of 0 and a range of [−1, 1]. A positive value indicates a positive affective state, while a negative value indicates a negative affective state. The initial value of C(k) is 0, which indicates the reinforcing effect of consecutive positive or negative emotions. This means that when the affective tendencies are the same between the two conversations, the value of C(k) increases and the degree of certainty increases; when the affective tendencies are different between the two conversations, the degree of certainty decreases. In the following, W(k) and C(k) are defined.

Knowledge graph ripple networks

A knowledge graph implements a knowledge mapping of the objective world from a string description to a structured semantic description, describing the rich facts and connections between entities in the objective world in the form of a graph structure. A generic representation of the knowledge graph is a triad, i.e., G = (H, R, T), where $\mathit{H}=\left\{e_1,e_2,\dots ,e_N\right\}$ is the set of head entities in the knowledge base and N is the number of entities; $\mathit{R}=\left\{r_1,r_2,\cdots ,r_M\right\}$ denotes the set of relationships in the knowledge base and M is the number of entity relationships; T denotes the set of tail entities in the knowledge base and $\mathit{T}\subseteq \mathit{H}\times \mathit{R}\times \mathit{H}$. $\mathit{H}$(head), $\mathit{R}$ (relationship), $\mathit{T}$ (tail) form a triad, i.e., head entity-relationship-tail entity.

In order to mine the potential content of interest to the participants using the knowledge graph, entity linking (De Cao et al., 2022; Yang & Liao, 2022) is used to extract and disambiguate the content of the participants’ conversations to obtain the set of conversational entities during the interaction, and then the set of entities is embedded into the knowledge graph ripple network. The relevant sets in the knowledge graph ripple network are defined as follows.

Based on the propagation model of the knowledge graph ripple network, the set of participant dialogue entities obtained in the kth dialogue is defined as

(8) $${\mathit{H}}^k=\left\{h^k|h^k\in \mathit{G}\right\}$$where G is a known knowledge graph, $h^k$is a dialogue entity, and k denotes the number of dialogue rounds. The triadic ripple set for the set ${\mathit{H}}^k$ of obtained participant dialogue entities is defined as

(9) $${\mathit{S}}_n^k=\left\{\left(h,r,t\right)\left|\left(h,r,t\right)\in \mathit{G}\right|h\in {\mathit{H}}^k\right\},n=1,2,\dots ,N$$where n indicates the entity at which level of association. For example, ${\mathit{S}}_1^1$ denotes the level 1 associated entity of the dialogue entity in dialogue 1.

The participant’s response set is then obtained through the retrieval formula, and the content of the response set is sequentially passed through the word2Vec method (Styawati et al., 2022) and the EmbeddingAverage vector averaging method (Chu et al., 2022) to obtain the sentence feature representation vector $v\in {\mathit{K}}^d$, where K is the feature representation vector and d is the word vector dimension. The probability $p_i$ of association of each triad $\left(h_i,r_i,t_i\right)$ with the sentence feature vector $v$ in the level 1 associated entity of the ripple set $S_1^k$ in the kth conversation is calculated as

(10) $$p_i=softmax\left({\mathit{v}}^T{\mathit{R}}_i{\mathit{h}}_i\right)={\displaystyle \frac{\exp \left({\mathit{v}}^T{\mathit{R}}_i{\mathit{h}}_i\right)}{\sum _{\left(h,v,t\right)\in {\mathit{S}}_1^k}exp\left({\mathit{v}}^T\mathit{R}\mathit{h}\right)}}$$where ${\mathit{R}}_i\in {\mathit{K}}^{d\times d}$, ${\mathit{h}}_i\in {\mathit{K}}^d$ denote the entity vectors of the ripple set $r_i$ and $h_i$ respectively. The entity vectors of the ripple set triple $\left(h_i,r_i,t_i\right)$ are obtained from the knowledge graph feature learning method TransD (Liu, Xie & Wang, 2017), corresponding to the entity vectors denoted as ${\mathit{h}}_i$, ${\mathit{R}}_{\mathit{i}}$ and ${\mathit{t}}_{\mathit{i}}$. The association probability $p_i$ can be viewed as the probability of measuring the similarity of a word vector $\mathit{v}$ to an entity ${\mathit{h}}_{\mathit{i}}$ in the relational ${\mathit{R}}_{\mathit{i}}$-space.

After obtaining the correlation probabilities, the influence of the dialogue entity on the level 1 associated entity tail entity $r$ is then calculated and the vector $o^1$ is calculated as

(11) $${\mathit{o}}^1=\sum _{\left(h_i,r_i,t_i\right)\in S_1^1}{p}_{i}t$$

The vector ${\mathit{o}}^1$ is the 1st order response of the sentence feature vector $\mathit{v}$, where ${\mathit{t}}_{\mathit{i}}\in {\mathit{K}}^d$. Through Eqs. (10) and (11), the content of interest to the participant is propagated farther along the hierarchical relationship of the knowledge graph.

Replacing the sentence feature vector $v$ in Eq. (10) with the vector ${\mathit{o}}^1$ in Eq. (11), we obtain the 2nd order response ${\mathit{o}}^2$, which is performed iteratively over the ripple set, i.e., the case where the values of $n$ in ${\mathit{S}}_n^k$ are 1, 2, respectively, representing the 2nd order propagation of the participant dialogue entity over the knowledge graph ripple network, because too large an order response dilutes the useful information, so the value of $N$ in Eq. (9) is either 2 or 3. In summary the participant’s response equation with respect to the vector $v$ has:

(12) $$\mathit{O}=\sum _{i=1}^H{\mathit{o}}^i$$

The normalized content probabilities are

(13) $$y=\sigma \left({\mathit{O}}^T\mathit{\nu }\right)$$

Define Eq. (13) as the participant content friendliness, where the sigmoid function is

(14) $$\sigma \left(x\right)={\displaystyle \frac{1}{1+\exp \left(-x\right)}}$$

Combining Eqs. (2) and (13), the content friendliness and sentiment friendliness weights are added to the candidate response content and normalized as

(15) $$y_{\nu }=\alpha R\left(k\right)+\beta y$$where $\alpha ,\beta$ is the constraint factor and has $\alpha +\beta =1$ (the default will be $\alpha =\beta =0.5$, as detailed later in the experimental discussion section). $y_v$ takes on a value range of [0, 1], with values closer to 1 indicating that the participant is more satisfied with the response.

Knowledge graph ripple network interaction model construction

Firstly, the knowledge graph $G$ is known (this article uses the Chinese general encyclopedic knowledge graph CN-Dbpedia, which is developed and maintained by the Knowledge Workshop Laboratory of Fudan University), the input of this article’s model is the $k$th participant interaction input content $C_H^k$, and the output is the $k$th+ $l$( $l$ > 1) robot’s reply content $C_R^{k+1}$; then, the continuous emotional state vector of the participant is obtained through the emotion friendliness Then, the continuous emotional state vector of the participant is obtained through the knowledge graph ripple network, while the content friendliness of the participant’s conversation is obtained through the knowledge graph ripple network; finally, the optimal reply of the robot for the conversation is given by considering the participant’s emotional state vector and the content friendliness together.

In each round of human-computer interaction, the emotional friendliness is obtained by matrix operations between the set of coordinates and positions in the emotional state space and the set of covariance matrices, and its time complexity is of constant order O(1); in the optimal entity content selection based on the knowledge graph ripple network, the number of dialogue content entities is of constant order and the number of entities involved in ripple propagation is also of constant order, then the ripple network propagation time complexity is of constant order $O\left(1\right)$; In the case of optimal entity content selection based on knowledge graph ripple networks, the number of dialogue content entities is constant and the number of entities involved in ripple propagation is also constant, then the time complexity of ripple network propagation is of constant order $O\left(1\right)$. Considering that the number of rounds of human-computer interaction is $n$, then the time complexity of this model is $O\left(n\right)$.

Settings

In order to conduct an effective experimental comparison of the HCI model proposed in this study, ChatterBot in Python is extended to the model proposed in this study and simulated in the form of a text chat. Figure 2 shows the workflow of the model in this study, where the solid part is the ChatterBot framework and the dashed part is the extension. Emotional friendliness is calculated by gestures, speech and facial expressions during human-computer interaction.

The used data were obtained from the dialogue corpus of the 2018 NLPCC task for Open Domain Question Answering, which is a Chinese dialogue question and answer corpus. The corpus has a total of 24,479 question-answer pairs, and 2,500 question-answer pairs are randomly selected as the validation set, another 2,500 question-answer pairs are randomly selected as the test set, and the remaining question-answer pairs are used as the model training set.

In this study, the following four models are selected for comparison.

(1) In Sutskever, Vinyals & Le (2014), the proposed model is a dialogue model for automatic response generation based on Seq2Seq (Sequence to Sequence) of LSTM (Long Short-Term Memory).

(2) In Gunther (2019), the proposed is a ChatterBot interaction model with answer ranking output based on confidence level.

(3) In Zhang, Wang & Mai (2018), the proposed is a cognitive model of dialogue with emotional MECs that takes into account the “empathy” of the participants during the interaction and selects emotionally similar responses as answers.

(4) In Lowe et al. (2015), the proposed is a ConceptNet cognitive model that stores general knowledge in an external memory module and integrates relevant general knowledge into a retrieval-based dialogue.

Evaluation metrics

The mean reciprocal rank (MRR) and mean average precision (MAP) were used to objectively evaluate the model response accuracy, where MRR reflects overall accuracy and MAP reflects single value accuracy, which are defined as

(16) $$MRR={\displaystyle \frac{1}{\left|k\right|}\mathbf{\sum }_{i=1}^k{\displaystyle \frac{1}{ran{k}_i^q}}}$$

(17) $MAP={\displaystyle \frac{1}{\left|k\right|}\sum _{i=1}^{k}Ave\left(A_i\right)}$

(18) $$A_{ve}\left(A_i\right)={\displaystyle \frac{\sum _{j=1}^n\left(r\left(j\right)/p\left(j\right)\right)}{n}}$$where, k represents the number of conversations that participants participate in, $ran{k}_i^q$ represents the rank of the i-th participant’s reply in the reply set, A_ve(A_i) represents the average accuracy of the reply sort of the i-th conversation model. p(j) represents the ranking level of the j-th answer in the adjusted reply set after the model adjusts the ordering of the candidate reply set considering the given limiting factors. r(j) represents the ranking of the j-th answer in the standard response set. n indicates the number of replies in the standard reply set. MAP reflects the average accuracy of the one-value accuracy of the recovery performance.

In order to further verify the effectiveness of the model, a manual evaluation method was adopted, and 40 volunteers were recruited according to the requirements to conduct interactive conversations with different cognitive model chatters. Firstly, the validity of each model is measured from the time dimension, and the duration of human-computer interaction is calculated. The volunteers then scored the fluency and sentiment evaluation criteria shown in Table 1.

Experimental results

In order to calculate the results of MRR and MAP two objective evaluation indexes, the number of standard response set was set as n = 10, that is, there were 10 candidate responses in the standard response set, and the calculation results were shown in Fig. 3. As can be seen from Fig. 3, compared with the other four comparison models, the proposed model achieves better results, mainly because the model proposed in this study considers both sentiment friendliness and content friendliness when ranking the candidate response set, which not only constrains the response content in terms of subjective sentiment friendliness, but also in terms of objective entity coherence. The MECs and ConceptNet models achieved better results when comparing the Seq2Seq and Chat-terBot models, as the MECs considered empathy in the interaction process and ConceptNet introduced external knowledge graphs as common-sense knowledge, both of which considered the sentiment and background knowledge factors respectively. The lowest Seq2Seq scores were due to the fact that many meaningless responses were generated during the interaction, taking into account factors such as security. The objective evaluation verifies that the model is effective in improving the response accuracy.

The manual evaluation took into account the influence of different age and gender groups on the evaluation of the interaction effect of the dialogue system. A total of 40 volunteers of different ages and genders were invited to interact with each model, and the number of rounds of interaction and the interaction time were counted. The results are shown in Fig. 4. A total of 20 male and 20 female volunteers were in each gender group (Group 1 and Group 2 respectively), and two female and two male volunteers were in each age group (19–22 years old for undergraduate students, 23–25 years old for master students, and 26–28 years old for social workers).

As shown in Fig. 4, compared with other models, volunteers performed better with this model in terms of the number of interaction rounds and interaction time; as shown in Figs. 5 and 6, volunteers in different gender and age groups scored this model higher than other models in Fluency and Sentiment. The human evaluation verifies that this model can effectively extend the number of rounds and duration of human-computer interaction by combining content coherence and emotional friendliness.

Experimental discussion

To further analyze the actual impact of sentiment friendliness and content friendliness on the model, a discussion of Eq. (15), i.e., the constraint a in $y_v=\alpha R\left(k\right)+{\beta }_y$, is carried out, as there is a relationship of $\beta =1-\alpha$, so only the impact of taking values of $\alpha$ between [0, 1] on the model needs to be discussed. The objective measures of MRR and MAP are still used to calculate the two indicators as a measure, and again the number of standard response sets $n=10$ is taken to calculate the two indicator values.

As can be seen from Figs. 7A and 7B, when $\alpha$ tends to zero, this model achieves better results in the objective evaluation of MAP and MRR compared to the ChatterBot model and the MECs model, but the difference is smaller compared to the Concept-Net model, because both this model and the ConceptNet model only consider content coherence under this constraint. The reason is that both the model and the ConceptNet model only consider content coherence under this constraint. As $\alpha$ converges to 1, the model achieves better results for MAP and MRR compared to the ChatterBot and ConceptNet models, but less so than the MECs model, because both the model and the MECs model only take into account emotional factors under this constraint. As the value of $\alpha$ converges to 0.5, this model achieves better results in both MAP and MRR objective ratings compared to the comparison models, as it takes into account both emotion and content friendliness, which can effectively improve the response accuracy.