A method for explaining individual predictions in neural networks

Basic idea

To understand the proposed method, we consider a simple neural network, as shown in Fig. 3. It comprises three input nodes and a single output node. The activation function for the output node is f(x) = x. In other words, the output value is the same as the weighted sum of the weights w and input x.

The output value y is calculated using the following equation:

(1) $$\begin{array}{rl}y& =w_1{x}_1+w_2{x}_2+w_3{x}_3\\ & =0.1\times 6+(-0.2)\times 6+0.4\times 10\\ & =0.6+(-1.2)+4\\ & =\mathrm{3.4.}\end{array}$$

Our goal is to determine the contribution of each input value to producing an output 3.4. As shown, the output value is the sum of w_ix_i and w_i is a fixed value. Therefore, we can consider w_ix_i to be the contribution of x_i to the output 3.4. We compute Cx_i, where Cx_i represents the contribution of x_i.

(2) $$C{x}_1=0.1\times 6=0.6$$ (3) $$C{x}_2=(-0.2)\times 6=(-1.2)$$ (4) $$C{x}_3=0.4\times 10=4$$ (5) $$y=\underset{i=1}{\overset{3}{\mathrm{\Sigma }}}C{x}_i=\mathrm{3.4.}$$

This calculation process provides important insights into the interpretation of the output y of the neural network. The baseline value of y is 0. Cx₁ and Cx₂ increase y to 4.6 (0.6 + 4), while Cx₃ decreases y by −1.2. As a result, the final value of y is 3.3. Owing to the different corresponding weight values, the contributions of x₁ and x₂ differ even though x₁ and x₂ have the same values. Here, we can observe the role of the weight values; they control the influence of each input value on the output. Based on the weight values, we can calculate the contribution of each input value to the output y.

Example 1

In the foundational idea presented above, we assume that the activation function $\varphi$ for the output node is $\varphi (x)=x$. However, an output node typically uses a specific activation function. In regression models, the “relu” activation function is generally used. Therefore, when calculating the contributions of the input values, the activation function should be considered. The neural network in Fig. 4 is the same as that in Fig. 3, except that the activation function is “relu.” If m >= 0 then relu(m) = m, else relu(m) = 0.

The activation function transforms the weighted sum into an output. Equations (2)–(4) include the transformation rates. Assuming that WS represents a weighted sum, output y is defined as y = relu(WS). The transformation rate (TR) is defined as TR = relu(WS)/WS = y/WS. The contribution of x_i is calculated using the following steps:

(6) $$\begin{array}{c}y=relu(WS)\\ =relu(3.4)\\ =3.4\hfill \end{array}$$

(7) $$\begin{array}{c}TR=y/WS\hfill \\ =3.4/3.4\hfill \\ =1.\hfill \end{array}$$

In Eq. (7), if WS is 0, then, we assign 0 to TR.

(8) $$\begin{array}{c}\mathrm{C}{x}_1=w_1\times x_1\times TR\hfill \\ =0.1\times 6\times 1=0.6\hfill \end{array}$$

(9) $$\begin{array}{rl}\mathrm{C}{x}_2& =w_2\times x_2\times TR\\ & =(-0.2)\times 6\times 1=(-1.2)\end{array}$$

(10) $$\begin{array}{c}\mathrm{C}{x}_3=w_3\times x_3\times TR\hfill \\ =0.4\times 10\times 1=4.\hfill \end{array}$$

Calculation of the contribution of x_i can be simplified using matrix operations. Let us denote the matrices x, W, and Cx, as well as the function diag(x):

$$x=\left[\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right],W=\left[\begin{array}{c}{w}_1\\ w_2\\ w_3\end{array}\right],Cx=\left[\begin{array}{c}C{x}_1\\ C{x}_2\\ C{x}_3\end{array}\right]$$

$$diag\left(\mathit{x}\right)=\left[\begin{array}{c}{x}_{1}00\\ 0x_{2}0\\ 00x_3\end{array}\right].$$

Then,

(11) $$\begin{array}{rl}Cx=& W^T\bullet diag\left(x\right)\times TR\\ =& \left[\begin{array}{c}{w}_1{x}_{1}TR\\ w_2{x}_{2}TR\\ w_3{x}_{3}TR\end{array}\right],\\ =& \left[\begin{array}{c}0.6\\ -1.2\\ 4\end{array}\right]\end{array}$$

$$\mathrm{y}=\underset{i=1}{\overset{3}{\mathrm{\Sigma }}}C{x}_i=3.4.$$

Example 2

The simple neural network in Example 1 has no hidden layers. In a deep neural network, computing the contribution of each input value to the final output is complex. Equation (11) must be calculated repeatedly for each layer. Consider the neural network shown in Fig. 5, which includes a hidden layer. The activation function for the hidden layer is “relu,” while the final output node uses the activation function f(x) = x.

For simple calculation, we pre-calculate TR:

(12) $$T{R}_{h1}=3.8/3.8=1//\mathrm{T}\mathrm{R}\mathrm{f}\mathrm{o}\mathrm{r}{h}_1$$

(13) $$T{R}_{h2}=0/-1.2=0//\mathrm{T}\mathrm{R}\mathrm{f}\mathrm{o}\mathrm{r}{h}_2$$

(14) $$T{R}_{out}=1//\mathrm{T}\mathrm{R}\mathrm{f}\mathrm{o}\mathrm{r}y.$$

We also define several matrices and a new operation denoted by ⊗:

$${\mathit{W}}_{\mathit{0}}=\left[\begin{array}{c}0.10.2\\ -0.20.1\\ 0.4-0.3\end{array}\right],{\mathit{W}}_{\mathit{1}}=\left[\begin{array}{c}0.4\\ 0.6\end{array}\right],\mathit{T}{\mathit{R}}_{\mathit{0}}=\left[\begin{array}{c}T{R}_{h1}\\ T{R}_{h2}\end{array}\right]=\left[\begin{array}{c}1\\ 0\end{array}\right]$$

$$\left[\begin{array}{c}{x}_{11}{x}_{12}{x}_{13}\\ x_{21}{x}_{22}{x}_{23}\end{array}\right]\otimes \left[\begin{array}{c}{z}_1\\ z_2\end{array}\right]=\left[\begin{array}{c}{x}_{11}{z}_1{x}_{12}{z}_1{x}_{13}{z}_1\\ x_{21}{z}_2{x}_{22}{z}_2{x}_{23}{z}_2\end{array}\right].$$

We first calculate Cx for h₁ and h₂ in Step 0.

(15) $$\begin{array}{rl}\mathit{C}{\mathit{x}}_{\mathit{S}\mathit{t}\mathit{e}\mathit{p}\mathit{0}}& ={{\mathit{W}}_{\mathit{0}}}^T\bullet diag\left(\mathit{x}\right)\otimes \mathit{T}{\mathit{R}}_{\mathit{0}}\\ & =\left[\begin{array}{c}0.1-0.20.4\\ 0.20.1-0.3\end{array}\right]\bullet \left[\begin{array}{c}600\\ 060\\ 0010\end{array}\right]\otimes \left[\begin{array}{c}1\\ 0\end{array}\right]\\ & =\left[\begin{array}{c}0.6-1.24\\ 1.20.6-3\end{array}\right]\otimes \left[\begin{array}{c}1\\ 0\end{array}\right]\\ & =\left[\begin{array}{c}0.6-1.24\\ 000\end{array}\right].\end{array}$$

In the next step, we calculate the final Cx, which is the same as Cx_Step1

(16) $$\begin{array}{rl}\mathit{C}{\mathit{x}}_{\mathit{S}\mathit{t}\mathit{e}\mathit{p}\mathit{1}}& ={{\mathit{W}}_{\mathit{1}}}^T\bullet \mathit{C}{\mathit{x}}_{\mathit{S}\mathit{t}\mathit{e}\mathit{p}\mathit{0}}\times \mathit{T}{\mathit{R}}_{\mathit{o}\mathit{u}\mathit{t}}\\ & =\left[0.40.6\right]\bullet \left[\begin{array}{c}0.6-1.24\\ 000\end{array}\right]\times 1\\ & =[0.24-0.481.6].\end{array}$$

From Eq. (16), we can determine the final output y and the contributions of the input values.

$$\begin{array}{rl}& {\mathrm{C}}_{x1}=0.24,{\mathrm{C}}_{x2}=-0.48,{\mathrm{C}}_{x3}=1.6\\ & \mathrm{y}=\sum _{i=1}^3{\mathrm{C}}_{x_i}=1.36.\end{array}$$

Equations (15) and (16) can be extended to deeper neural networks using a feedforward approach.

Algorithm for proposed method

In this section, we describe the formal NNexplainer algorithm for calculating the contributions of the input values. For simplicity, we assume the following functions:

diag(): transform a list to diagonal matrix

mat_mul(): matrix multiplication of two matrix

list_mul(): multiply corresponding elements of two list and form a new list

list_sum(): summate all elements of a list

To aid in understanding Algorithm 1, an example of the actual calculation process is provided in the Supplemental Material.

Implementation and test

We implemented the proposed NNexplainer using Python 3.12 and its related packages, as listed in Table 1.

To test the implemented NNexplainer, we selected the Boston housing dataset (Harrison & Rubinfeld, 1978, https://www.kaggle.com/datasets/heptapod/uci-ml-datasets). This dataset is widely used to build example regression models, with the goal of predicting the exact house price. It contains fourteen variables (features), and “medv” is the target variable representing the house price. Table 2 presents these variables.

We built a regression model to predict “medv” based on the Boston housing dataset. Table 3 presents the correlation coefficients between “medv” and other variables. The variable “rm” exhibits a strong positive correlation with house prices, whereas “lstat” shows a strong negative correlation.

The architecture of the model is illustrated in Fig. 6, where the numbers in parentheses represent the number of nodes. The learning parameters of the model are listed in Table 4. The entire dataset is split into training and test sets, which are then standardized to a range of 0 to 1.

Reliability of the proposed method

As mentioned earlier, to calculate the contribution of each input variable to the prediction result, the proposed method follows the basic calculation mechanism of a feedforward neural network. In a feedforward neural network, input values move unidirectionally from the input layer to the output layer. Beyond this basic mechanism, the proposed method preserves the contribution ratio of each input value across all nodes in the hidden layer until it reaches the output node. In other words, the list of contribution ratios flows through the network and reaches the output node without exception. If the neural network is well-designed, the proposed method consistently determines the contribution of input variables to the prediction result.

In deep neural networks, the proposed method functions effectively. During the calculation process, interruptions such as the ‘vanishing gradient’ phenomenon do not occur. This is because, as seen in lines 18 to 35 of Algorithm 1, the contribution ratio list is updated based on the output values of each hidden layer. As long as all output values of the hidden layer are not zero, the contribution ratio list updates normally.

In conclusion, the proposed method is reliable and can be generally applied to all types of feed forward neural networks, including deep neural networks. However, additional research is needed for recurrent neural network (RNN) or convolutional neural network (CNN).

Comparison with LIME and SHAP

To compare with LIME and SHAP, we generated LIME and SHAP plots, as shown in Figs. 11 and 12, using the Python “lime” and “shap” packages. The input feature values listed in Table 5 were tested. For clarity, we reformatted the original LIME plot using the same result data from the “lime” package. The displayed SHAP plot is a “shap waterfall” plot. In both figures, the input values are standardized versions of the original values.

In the LIME plot presented in Fig. 11, the features represented by blue bars have a positive influence on the prediction, whereas those represented by red bars have a negative influence. The values on the x-axis indicate the relative magnitude of the influence on the prediction. In the SHAP plot shown in Fig. 12, the red arrow indicates an increase in house prices, whereas the blue arrow indicates a decrease. In the SHAP model, the expected house price, expressed as E(f(x)) is 23.177. Each feature value can either increase or decrease the expected house price. For example, the value of “lstat” increases the expected house price by 2.14, whereas the value of “indus” decreases it by 1.02.

The contribution values displayed in the plots from LIME, SHAP, and the proposed NN explainer have different meanings; therefore, caution is necessary when interpreting these plots. The disadvantages of LIME and SHAP are discussed in the introduction. The advantages of NNexplainer over LIME and SHAP are as follows:

• –

  NNexplainer is both simple and fast, requiring only feedforward computation for the input values. It does not require additional information beyond the weights of the trained prediction model. By contrast, LIME requires building a local interpretation model, whereas SHAP requires Monte Carlo sampling and computation of Shapley values.

• –

  NNExplainer provides a unique and stable explanation for the given input values. The explanatory results remain consistent unless the prediction model is modified. By contrast, the explanatory results from LIME and SHAP can vary depending on the local interpretation model or the outcomes of Monte Carlo sampling.

• –

  The contribution value for each input is absolute, similar to that of SHAP, and its meaning remains consistent across different prediction models. By contrast, LIME provides only relative contribution values.

• –

  NNexplainer is easier to understand than LIME and SHAP. If you understand neural networks, you can also understand NNexplainer.

Proposed method for classification model

We have discussed the proposed method from a regression perspective. However, it also applies to classification models produced by neural networks. In a neural network, the classification models have N output values, where N represents the number of classes, and the regression models yield a single output. Consequently, the meaning of “contribution” for each feature value differs in the proposed method. In regression models, each feature value contributes to the formation of the output value. By contrast, in classification models, each feature value supports or opposes each class. Consequently, the contribution plot for the classification model differs from that of the regression model. Figure 13 shows an example contribution plot for the classification model. In this task, there are three classes, and the input case is predicted to belong to Class 2. The lengths of the negative and positive side bars indicate the degree of opposition and support, respectively. For example, the value of “proline” opposes both Class 1 and Class 2 while supporting Class 0. Class 0 is strongly supported by the values of “proline,” “c_intensity,” and “magnesium,” but is also strongly opposed by the values of “n_phenols” and “al_of_ash.” Consequently, the overall degree of support is low. The prediction model selects the class with the highest support for prediction. Figure 13 illustrates the influence and direction of each feature value during the prediction process.

To obtain the contribution plot in Fig. 13, we slightly modified Algorithm 1. Lines 25–32 are not executed when layer (ln) is the final layer. In a classification model, the typical activation function for the last layer is “softmax,” which transforms weighted sums into probability values ranging from 0 to 1. Consequently, the output values become very small, making it challenging to observe the influence of each feature value on the class output values.

Proposed method as a surrogate model

The proposed NNexplainer demonstrates that regression and classification models based on neural networks are no longer black boxes. Consequently, we can explain the predictions of the black box models using neural networks and the proposed method. In other words, neural networks can serve as surrogate models to elucidate black box models. To illustrate this, we constructed a random forest (RF) regression model using the Boston housing dataset and obtained the predicted values of y_pred from the model. Subsequently, we trained a neural network model using the training set and y_pred to simulate the original RF model. The correlation coefficient between y_pred and the predictions of our surrogate model is approximately 0.97, indicating that the surrogate model effectively simulates the original RF model. Figure 14 shows the contribution plots for the input values listed in Table 5. Despite using the same input values, Figs. 8 and 14 are notably different. For instance, in the RF model, the value of “rm” decreases the house price, whereas the same value increases it in Fig. 8. This discrepancy suggests that the prediction rules or mechanisms differ between neural network and RF models. As demonstrated, the proposed method allows the neural network model to serve as a surrogate for other black box models.

Further discussion

According to Algorithm 1, the time required to generate an explanation for a prediction result in the proposed method is proportional to the time needed to produce the prediction itself. The time required to generate a prediction in a neural network model depends on the number of hidden layers and the number of nodes in each hidden layer. Since these parameters can be arbitrarily designed, the exact time to generate a prediction result cannot be determined precisely. However, assuming the number of layers is M and each layer contains N nodes, the time complexity for generating an explanation of the prediction result is O(N²·(M-1)).

The proposed model has broad applicability across various domains. For instance, in a neural network model designed to detect fraudulent transactions, it can help users understand the rationale behind classifying a transaction as fraudulent, making it easier to review or adjust decisions when necessary. Similarly, in a loan applicant credit risk assessment model, lenders can make more informed and equitable lending decisions, even for individuals with low credit scores. In the medical field, when applied to a disease diagnosis model, the approach can provide a clear basis for identifying patients, enhancing transparency and trust in the diagnostic process.