Multiple imputation methods: a case study of daily gold price

Mean imputation

In mean imputation, a missing value for a particular variable is replaced by the average of currently available examples. This strategy is simple and does not change the sample size, but the estimates of standard deviation and variance are likely to be low because the strategy reduces the amount of variation in the data. Variation reduction also decreases the correlations and covariances, so this technique often produces incorrect estimations regardless of the underlying explanation for missing data (Donders et al., 2006).

The procedure and the mathematical framework for mean imputation are as follows:

1. Identify the missing value

   If we have a dataset with nobservations for a particular variable X, where X_i represents the i − th observation. Some of these observations might be missing.

2. Calculate the mean of the observed values

3. Calculate the mean $\overline{X}$of the non-missing values in the variable X using Eq. (1): (1)${\displaystyle \begin{array}{c}\hfill \overline{X}=\frac{1}{n-m}\sum _{i=1}^n{X}_i\hfill \end{array}}$ where m is the number of missing values in X.

4. Impute the missing values Replace each missing value with the calculated mean $\overline{X}$ using Eq. (2): (2)${\displaystyle \begin{array}{c}\hfill X_{imputed,i}=\left\{\begin{array}{c}{X}_{i}if{X}_i,isobserved\hfill \\ \overline{X}if{X}_i,ismissing.\hfill \end{array}\right.\hfill \end{array}}$ Here, X_imputed,i represents the value of X_i after imputation.

Hot-deck imputation

Hot-deck imputation was extensively used in the past. In this approach, a missing value is substituted by the value of a record that was randomly selected and close to the value of the original record. The term “hot deck” was coined at the time when data were stored on punched cards. It means that the information donors are from the same dataset as the information receivers. The term hot stack of cards is used because it refers to being worked on just at that moment (Christopher et al., 2019).

Last observation carried forward (LOCF) is one of the methods of hot-deck imputation. First, LOCF produces an ordered dataset by sorting a dataset on the basis of any one of numerous factors. Second, the approach finds the first missing value and imputes it by using the value of the cell that was present immediately before the missing value. The technique is applied repeatedly for each cell that contains a missing value until all of the missing values have been estimated. This technique resembles the opinion that if a measurement for a person or an entity is absent, the ideal assumption is that it has not changed from the previous time it was obtained. This situation is common in cases where the instances are repeated measurements of a variable for the same person or object. In other words, if a measurement is absent, the most reasonable assumption is that it has remained unchanged from the time it was last measured. However, the use of this strategy increases the likelihood of obtaining incorrect conclusions and contributes to an increase in bias (Mavridis et al., 2019; Wongoutong, 2021).

The procedure and the mathematical framework for hot-deck Imputation are as follows:

1. Define the dataset

   • Let $\mathrm{X}=\left(x_{ij}\right)$ be an n × p matrix of observed data, where n is the number of observations and p is the number of variables.

   • Let $\mathrm{M}=\left(m_{ij}\right)$be an n × pmatrix of missing data indicators, where m_ij = 1 if x_ij is missing, and m_ij = 0 otherwise.

2. Identify donors and recipients

   • Define the set of donors 𝔇 andthe set of recipients ℜ: ${\displaystyle \mathfrak{D}=\left\{i\mid m_{i}j=0\setminus \text{for all}j\right\}}$ ${\displaystyle \mathfrak{R}=\left\{i\mid m_{ij}=1\setminus \text{for some j}\right\}}$

3. Determine similarity

   • Define a similarity measure $\mathrm{S}\left(i,k\right)$ between recipient i anddonor k. Common choices include Euclidean distance, Mahalanobis distance, or other distance metrics.

4. Select the donor

   • For each missing value x_ij ($\text{where}\left(\mathrm{i}\in \mathfrak{R}\right)$), select a donor k($\text{where}\left(\mathrm{k}\in \mathfrak{D}\right)$) such that $\mathrm{S}\left(i,k\right)$ isminimized. This means finding the donor that is most similar to the recipient in terms of observed data.

5. Impute the missing value

   • Replace the missing value x_ij with the corresponding value from the selected donor k: ${\displaystyle x_{ij}=x_{kj}.}$

KNN imputation

KNN is a nonparametric supervised learning approach. It was introduced in 1951 by Evelyn Fix and Joseph Hodges for the field of statistics, and Thomas Cover developed it further. KNN is employed in regression and classification. The input is made up of k training cases in a data collection that are the most similar to one another under both instances (regression and classification). Whether KNN is utilized for classification or regression decides the result (Sanjar et al., 2020).

Class membership is the outcome of KNN classification. How the object should be categorized is voted by a plurality of an item’s neighbors. Then, the object is placed in the class that has the highest frequency among the item’s k closest neighbors (k is a positive integer and often small). The item in question is viewed as belonging to the category of its one closest neighbors when k is equal to 1. KNN regression generates the value of a certain attribute associated with an object, and this value is computed by obtaining the average of the values of the k closest neighbors (Sanjar et al., 2020). In KNN classification, the function is only approximated locally, and the entire computation is postponed until the function is assessed. If the features represent different physical units or have considerably different scales, then training data normalization can remarkably improve the accuracy of the algorithm because the classification of this algorithm is based on distance. Notably, the distance between features can exhibit considerable variation (Bania & Halder, 2020).

The assignment of weights to the contributions provided by neighbors may be a good classification and regression strategy. It ensures that the influence of neighbors situated close to the center of the map on the general average is greater than the influence of neighbors located far from the map’s center. For example, in a common technique of weight calculation, each neighbor is assigned the number 1/d as its weight; d is the distance that separates the subject and the neighbor (Jadhav, Pramod & Ramanathan, 2019).

In KNN classification or KNN regression, the neighbors are selected from a set of objects in which the class (for KNN classification) or the object (for KNN regression) attribute value has already been determined. Although an explicit training phase is unnecessary, this set may still be regarded as the algorithm’s training set (Jadhav, Pramod & Ramanathan, 2019).

1. Identify the missing values:

   • Let X be the dataset with nsamples and features. Identify the missing value(s) in X. Suppose x_i,j is the missing value in the i − th sample and j − th feature.

2. Calculate distances

   • For each sample i with a missing value, calculate the distance to all other samples that have a known value in the j − th feature. A common distance metric used is the Euclidean distance. For two samples pandq, the Euclidean distance is given by Eq. (3): (3)${\displaystyle \begin{array}{c}\hfill d\left(p,q\right)=\sqrt{\sum _{k=1}^m{\left(x_{p,k}-x_{q,k}\right)}^2}\hfill \end{array}}$ where the summation is over all features kfor which x_p,k and x_q,k are not missing.

3. Select nearest neighbors

   • Choose the K nearest neighbors to the sample i based on the calculated distances. Let these neighbors be indexed by $\mathrm{N}\left(\mathrm{i}\right)$.

4. Impute the missing value

   • There are different strategies to impute the missing value using the nearest neighbors:

   1. Mean imputation using Eq. (4) (4)${\displaystyle \begin{array}{c}\hfill x_{i,j}=\frac{1}{K}\sum _{q\in N\left(i\right)}{x}_{q,j}\hfill \end{array}}$

   2. Median imputation using Eq. (5) (5)${\displaystyle \begin{array}{c}\hfill x_{i,j}=\text{median}{x}_{q,j}:\mathrm{q}\in N\left(i\right)\hfill \end{array}}$

Random forest imputation

One of the ensemble learning techniques is random forest. Throughout the training phase, this technique creates many decision trees, and it consolidates the forecasts generated by separate decision trees. The random forest method’s initial stage entails the construction of decision trees. A random subset of the training data is used to build each decision tree inside the random forest. A randomly selected subset of characteristics is evaluated for splitting at every node of the decision tree. Such randomization reduces the correlation between individual trees and mitigates the overfitting problem. In the process of training, each decision tree is constructed by randomly selecting (with replacement) a portion of the training data. A subset of characteristics for every node in a decision tree is then randomly chosen. The selected characteristics are utilized to identify the optimal division at that particular node (e.g., the use of Gini impurity for classification or mean squared error for regression). The procedure is continuously repeated until the tree has reached its maximum size (or until a condition for stopping has been met). In the process of prediction, the input characteristics are made to pass through each decision tree in the forest (the third phase). In classification tasks, each tree votes for a certain class, and the class with the most votes is chosen as the ultimate prediction. In regression tasks, each tree continuously generates value predictions, and the final prediction is obtained by taking the average or median of these forecasts. Aggregation is the final step in the random forest process. During aggregation, the aggregated forecast is acquired by combining the predictions from all the individual trees. In classification tasks, this process may involve determining the mode that represents the most common prediction. In regression analysis, the average or median value of the predicted outcomes is calculated (Ren et al., 2023).

The process of using random forest for imputation involves the following steps:

1. Initial imputation:

Fill the missing values with initial guesses, often using mean/mode for numerical/categorical variables.

1. Iterative imputation:

2. For each feature j with missing values, separate the dataset into two parts: D_obs (where j is observed) and D_miss (where j is missing).

3. Train a random forest model 𝔉_𝔧 on D_obs using all other features to predict j.

4. Predict the missing values in D_miss using 𝔉_𝔧.

5. Replace the initial guesses of the missing values in 𝔉_𝔧 with the predictions from 𝔉_𝔧.

1. Repeat

Iterate over all features with missing values, updating the imputations until the imputations converge or a set number of iterations is reached

• Mathematical formulation

Given a dataset X with n samples and p features, let x_ij denote the value of feature j for sample i. Let X_¬j represent all features except j.

1. Initial imputation:

${\hat{X}}_{ij}^{\left(0\right)}$ = initial guess for missing value.

1. Iterative imputation: For iteration t: (6)${\displaystyle \begin{array}{c}\hfill {\hat{x}}_{ij}^{\left(t\right)}=F_{j^{\left(t\right)}}\left(x_{i,\neg j}\right).\hfill \end{array}}$ (7)${\displaystyle \begin{array}{c}\hfill x_{ij}^{\left(t+1\right)}={\hat{x}}_{ij}^{\left(t\right)}.\hfill \end{array}}$

   1. For each feature j:

      1. Train Random Forest ${\mathrm{F}}_{{\mathrm{j}}^{\left(\mathrm{t}\right)on}}\left\{\left(x_{i,\neg j},x_{ij}\right)\mid x_{ij\text{i osre}}\right\}$.

      2. Predict missing values using Eq. (6):

   1. Update the imputed values using Eq. (7):

2. Convergence: Continue until convergence or maximum iterations.

Spline imputation

The interpolant in spline imputation is a particular kind of piecewise polynomial known as a spline. Spline imputation is a type of interpolation. Spline interpolation fits nine cubic polynomials between each pair of 10 points instead of fitting a single 10° polynomial to all the points. This approach is different from the traditional high-degree linear interpolation method that fits a single high-degree polynomial to all the points simultaneously. Spline interpolation is often preferred over polynomial interpolation because the former’s interpolation error is small even when low-degree polynomials are used for the spline. Spline interpolation is employed in various applications. Runge’s phenomenon, in which oscillation occurs between points during interpolation by high-degree polynomials, is prevented by spline interpolation (Tayebi, Momani & Arqub, 2022).

The process of using Spline imputation involves the following steps:

Given a set of observed data points $\left(x_1,y_1\right),\left(x_2,y_2\right),\dots ..,\left(x_n,y_n\right)$ with some missing values, spline interpolation aims to fit a smooth curve $f\left(x\right)$ as in Eq. (8): (8)${\displaystyle \begin{array}{c}\hfill f\left(x_i\right)=y_i\text{for all observed}\left(x_i,y_i\right).\hfill \end{array}}$

The most common spline used is the cubic spline, which is a piecewise cubic polynomial. The cubic spline $S\left(x\right)$ is defined in Eq. (9): (9)${\displaystyle \begin{array}{c}\hfill S\left(x\right)=\left\{\begin{array}{cc}{a}_1+b_1\left(x-x_1\right)+c_1{\left(x-x_1\right)}^2+d_1{\left(x-x_1\right)}^3\hfill & forx\in \left[x_1,x_2\right]\hfill \\ ⋮\hfill & ⋮\hfill \\ a_{n-1}+b_{n-1}\left(x-x_{n-1}\right)+c_{n-1}{\left(x-x_{n-1}\right)}^2+d_{n-1}{\left(x-x_{n-1}\right)}^3\hfill & forx\in \left[x_{n-1},x_n\right]\hfill \end{array}\right.\hfill \end{array}}$

• Conditions for cubic splines

Continuity:

The spline $S\left(x\right)$ must be continuous at each data point x_i. ${\displaystyle S\left(x_i\right)=y_i\text{for all}i}$

Smoothness:

The first and second derivatives of the spline must be continuous at each internal data point. ${\displaystyle S^{{}^{\prime }}\left(x_i^{-}\right)=S^{{}^{\prime }}\left(x_i^{+}\right)\mathrm{and}{S}^{{}^{\prime \prime }}\left(x_i^{-}\right)=S^{{}^{\prime \prime }}\left(x_i^{+}\right)\text{for all internal}{x}_i.}$

• Boundary conditions: There are several choices for boundary conditions, but common ones include:

Natural spline:

The second derivative at the endpoints is zero. ${\displaystyle S^{{}^{\prime \prime }}\left(x_1\right)=0\mathrm{and}{S}^{{}^{\prime \prime }}\left(x_n\right)=0.}$

Clamped spline:

The first derivative at the endpoints is specified. ${\displaystyle S^{{}^{\prime }}\left(x_1\right)=f^{{}^{\prime }}\left(x_1\right)\mathrm{and}{S}^{{}^{\prime }}\left(x_n\right)=f^{{}^{\prime }}\left(x_n\right).}$

Solving for coefficients

To find the coefficients (a_i, b_i, c_i, d_i) for each segment, we set up a system of linear equations based on the conditions above and solve for the unknowns.

SVM imputation

SVM is widely recognized in data imputation because it can properly address high-dimensional data and understand intricate connections within the data. SVM identifies the advantageous hyperplane that divides distinct classes or data points in a space with several dimensions. SVM may be utilized to predict missing values in data imputation because it can identify patterns and connections from available data (Dudzik, Nalepa & Kawulok, 2024).

The SVM process involves different stages. At the feature extraction stage, the dataset is expressed as a set of distinct characteristics or variables. These attributes characterize each observation or data point. At the training stage, the SVM algorithm learns from the entire observed data and determines the most advantageous among hyperplanes to effectively distinguish between several classes or data points in the feature space. At the imputation stage, the SVM model is trained and subsequently used to predict missing values in the dataset on the basis of the associations learned at the training stage. The values of other characteristics in the dataset are utilized by the model to forecast the missing values. At the evaluation stage, the correctness and consistency of the imputed values are assessed. The quality of the imputation is evaluated using various metrics and methodologies, including MSE or cross validation. At the integration stage, the gaps in the data are filled by incorporating the inferred values into the dataset (Dudzik, Nalepa & Kawulok, 2024).

The process of using SVM imputation involves the following steps:

1. Problem definition given a dataset $X=\left\{\left(x_1,y_1\right),\left(x_2,y_2\right),\dots ,\left(x_n,y_n\right)\right\}$ with missing values, the goal is to predict the missing values y_i for the corresponding x_i.

2. Data preparation

   • Feature matrix: X is the feature matrix where each row represents an instance and each column represents a feature

   • Target Vector: y is the target vector containing the values to be predicted. Some entries in yare missing.

3. Handling missing data

   • Separate the data into two parts: observed data $\left(X_{\mathrm{obs}},y_{\mathrm{obs}}\right)$ and missing data $\left(X_{\text{miss}},y_{\text{miss}}\right).$

   • X_obsandy_obs represent the complete cases, while X_miss and y_miss represent the incomplete cases where y_miss needs to be predicted.

4. Training the SVM model (10)${\displaystyle \begin{array}{c}\hfill \underset{w,b,\xi ,{\xi }^{\ast }}{min}\frac{1}{2}|w{|}^2+C\sum _{i=1}^n\left({\xi }_i+{\xi }_i^{\ast }\right)\hfill \end{array}}$

   • Objective function: The SVM aims to find a hyperplane that best separates the data into classes (for classification) or fits the data (for regression).

   • Support vectors: These are the data points that are closest to the hyperplane and determine its position.

   • Kernel function: $K\left(x_i,x_j\right)$transforms the input data into a higher-dimensional space to make it easier to find a separating hyperplane.

   • Optimization problem: (for regression) using Eq. (10):

subject to Eqs. (11) and (12) (11)${\displaystyle \begin{array}{c}\hfill y_i-\left(w\cdot \varphi \left(x_i\right)+b\right)\le ϵ+{\xi }_i\hfill \end{array}}$ (12)${\displaystyle \begin{array}{c}\hfill \left(w\cdot \varphi \left(x_i\right)+b\right)-y_i\le ϵ+{\xi }_i^{\ast }\hfill \end{array}}$ ${\xi }_i,{\xi }_i^{\ast }\ge 0$

1. Predicting missing values

   Use the trained SVM model to predict the missing values y_miss for the incomplete cases X_miss using Eq. (13): (13)${\displaystyle \begin{array}{c}\hfill \widehat{y_i}=w\cdot \varphi \left(x_i\right)+b.\hfill \end{array}}$ The predicted values $\left(\widehat{y_i}\right)$ are the imputed values for the missing entries in y_miss.

ME

The average of all the individual mistakes in a collection is called ME. Measurement uncertainty or the disparity between measured and real values is referred to as an error in this context. Error is formally called measurement or observational error (Karunasingha, 2022). Equation (14) can be used to derive ME. (14)${\displaystyle \begin{array}{c}\hfill ME=\frac{\sum _{i=1}^n{y}_i-x_i}{n}.\hfill \end{array}}$

MAE

The differences in the results derived from two observations that exhibit similar phenomena is measured by MAE. Examples of Y versus X include comparisons of initial time with future time, one measurement method versus another measurement technique, and one measurement technique versus an alternate measurement technique. MAE is calculated as the total number of absolute errors divided by the number of samples (Ji & Peters, 2003). Equation (15) shows the derivation of MAE. (15)${\displaystyle \begin{array}{c}\hfill MAE=\frac{\sum _{i=1}^n\left|y_i-x_i\right|}{n}.\hfill \end{array}}$

RMSE

RMSE refers to the difference between actual and expected values given as a standard deviation value (prediction error). The residuals measure how far the data points deviate from the regression line, and RMSE shows how distributed these residuals are. Both measurements are referred to as RMSE. RMSE provides indicates the extent to which the data are centered around the best-fit line. RMSE is a common method to confirm the accuracy of experimental data in the fields of climatology, forecasting, and regression analysis (Karunasingha, 2022). RMSE can be expressed as in Eq. (16). (16)${\displaystyle \begin{array}{c}\hfill RMSE=\sqrt{\frac{\sum _{i=1}^n{\left(y_i-x_i\right)}^2}{n}}.\hfill \end{array}}$

MPE

MPE is the average of percentage errors. It determines the extent of the difference between forecasts of a model and the actual values of the quantity being forecasted (Karunasingha, 2022). Equation (17) shows the derivation of MPE. (17)${\displaystyle \begin{array}{c}\hfill MPE=\frac{100\text{\%}}{n}\sum _{i=1}^n\frac{y_i-x_i}{y_i}\hfill \end{array}}$

MAPE

In statistics, MAPE measures the prediction accuracy of a forecasting approach. Accuracy is often expressed as a ratio in MAPE (Awajan, Ismail & AL Wadi, 2018; Karunasingha, 2022). Equation (18) shows how MAPE is obtained. (18)${\displaystyle \begin{array}{c}\hfill MAPE=\frac{100\text{\%}}{n}\sum _{i=1}^n\left|\frac{y_i-x_i}{y_i}\right|\hfill \end{array}}$