Classifying reservoir facies using attention-based residual neural networks

Transformer

A Transformer consists of a multi-head self-attention layer followed by a position-wise feed-forward layer, with element-wise addition and layer normalization being done after each layer (Vaswani et al., 2017). A self-attention layer comprises three parametric matrices Key, Query, and Value. Each input embedding is projected onto these matrices, to generate their Key, Query, and Value vectors. Formally, let $K\in {\mathbb{R}}^{m\times k},Q\in {\mathbb{R}}^{m\times k}$ and $V\in {\mathbb{R}}^{m\times v}$ be the matrices comprising key, query and value vectors of all the embeddings, respectively, and $m$ be the number of embeddings inputted to the Transformer, $k$ and $v$ be the dimensions of the key and value vectors, respectively. Every input embedding attends to all other embeddings through an Attention head, which is computed as follows:

(1) $$Attention(K,Q,V)=softmax\left(\frac{Q{K}^T}{\sqrt{k}}\right)\cdot V.$$

This operation allows each feature to attend to all other features, weighting their contributions based on relevance, thereby capturing both short- and long-range dependencies. In well-log data, where features such as resistivity, porosity, and gamma-ray measurements interact in complex, non-sequential ways, this attention mechanism facilitates feature interaction modeling without relying on strict positional order. We use multi-head attention, which allows the model to learn multiple, complementary relationships between features in parallel. The output is passed through a feed-forward network (FFN) consisting of two linear layers with ReLU activation, followed by residual addition and layer normalization:

(2) $$FFN(x)=ReLU(x{W}_1+b_1)\cdot W_2+b_2.$$The residual structure and normalization facilitate gradient flow and prevent feature drift during training, enhancing model robustness.

Residual block

The Residual block is designed to learn effective and robust numerical features, taking inspiration from the TabNet architecture (Arik & Pfister, 2019) for tabular data processing (Arik & Pfister, 2019). The block consists of two sequential layers, where each FC layer is followed by a BatchNorm and a GLU activation function (Dauphin et al., 2016). A residual connection combines the input with the final output after the GLU layer, helping to stabilize the learning process by maintaining consistent variance throughout the network (Gehring et al., 2017). The first FC layer expands the feature dimension from 7 to 64, while the second FC layer maintains the 64-dimensional representation with a $64\times 64$ transformation. The detailed architecture is as follows:

• Input dimension: ${\mathbb{R}}^7$ (e.g., 7 numerical well-log features)

• FC1: Expands the input to ${\mathbb{R}}^{64}$

• BatchNorm1 $\to$ GLU1 (outputs ${\mathbb{R}}^{64}$)

• FC2: ${\mathbb{R}}^{64\times 64}$ transformation

• BatchNorm2 $\to$ GLU2

A residual connection adds the original input (after projection if needed) to the output of the second GLU activation:

(3) $$Output=GLU2(BN2(FC2(GLU1(BN1(FC1(x))))))+x.$$

The use of residual connections in the network enhances gradient stability and ensures a consistent feature scale across various layers, simplifying the training of deep networks. Incorporating BatchNorm layers helps reduce internal covariate shift, speeding up convergence and making the network less sensitive to the initial settings of its parameters. Furthermore, implementing GLUs instead of traditional activation functions like ReLU has proven effective for structured data. This method offers improved gating dynamics and feature sparsity, ultimately leading to better interpretability and generalization of the model. Together, these design choices enable the Residual block to serve as a stable and expressive numerical encoder in the overall dual-stream architecture.

Feature representation

Let $(x,y)$ denote a feature-target pair, where $x=(x_{cat},x_{cont})$. Here, $x_{cat}$ represents all categorical features and $x_{cont}\in \mathbb{R}$ represents continuous features. Let $x_{cat}=(x_1,x_2,x_3,\cdots ,x_m)$, where each $x_i$ is a categorical feature for $i\in 1,\cdots ,m$. Each categorical feature $x_i$ is encoded into a vector with dimension $d$ using a Transformer layer. Let $T_{cat}\in {\mathbb{R}}^d$ for $i\in 1,\cdots ,m$ represent the encoded features obtained after the Transformer block, computed as:

(4) $$T_{cat}=\mathcal{T}(x_{cat})=(\mathcal{T}(x_0),\mathcal{T}(x_1),\mathcal{T}(x_2),\cdots ,\mathcal{T}(x_m)),$$where $T_{cat}$ is the set of embeddings for all categorical features and $\mathcal{T}$ denotes the operation of the Transformer block. The numerical features $x_i$ are processed by a normalization layer before being forwarded to the residual block as follows:

(5) $$R_{cont}=(\mathcal{R}(h_0),\mathcal{R}(h_1),\mathcal{R}(h_2),\cdots ,\mathcal{R}(h_n)),$$where $h_i\in \mathbb{R}$ represents the continuous features, $\mathcal{R}$ denotes the normalization process, and $R_{cont}\in {\mathbb{R}}^n$ is the set of encoded features obtained after the Residual block for $i\in 1,\cdots ,n$. The encoded categorical and numerical vectors are then concatenated into a unified vector $F_{concat}$ with a dimension of 128, described as:

(6) $$Fconcat=\left(\begin{array}{c}{T}_{cat}\\ R_{cont}\end{array}\right)=\left(\begin{array}{c}{x}_1\\ x_2\\ ⋮\\ x_m\\ h_1\\ h_2\\ ⋮\\ hn\end{array}\right)$$

This vector is finally passed to the MLP layer, denoted by $G_{\psi }$, to predict the target $y$. Cross-entropy is used as the loss function for the classification tasks.

Experimental setup

Our models were trained for 50 epochs using a learning rate of 0.01, a batch size of 64, and the Adam optimizer (Kingma & Ba, 2014). The experiments were conducted using PyTorch framework version 2.0.0. The DL models were executed on an RTX 3090 GPU with 24 GB of memory. Data loading and processing were performed on a computer equipped with an AMD Ryzen 7 5800X 8-Core Processor (3.80 GHz) running Windows 11 Professional.

Data preprocessing

The dataset comprised well-log records from eight wells, representing a range of geological settings. Each record included both numerical features (such as gamma-ray, resistivity, and porosity) and categorical attributes (like lithology indicators). Numerical features were standardized using z-score normalization to achieve a mean of zero and a variance of one across all features. Categorical variables were transformed into dense vector representations through learnable embedding layers optimized jointly during training. Missing values in the numerical features were imputed using the mean of each feature while missing categorical values were represented by a dedicated ‘unknown’ embedding token.

Evaluation metrics

Model performance was evaluated using two primary metrics:

• Area under the receiver operating characteristic curve (AUROC): This metric evaluates the model’s ability to distinguish between facies classes at different decision thresholds. It is particularly effective for assessing classification performance when class distributions are balanced or moderately imbalanced.

• Area under the precision-recall curve (AUPRC): This metric assesses the balance between precision and recall, making it particularly useful in situations of class imbalance. Given that some facies types are underrepresented in the dataset, AUPRC offers a more accurate measure of the model’s ability to identify minority classes.

These metrics and accuracy (ACC) and F1-score (F1) were calculated on the held-out test set. The final results reported represent the average performance across five independent training runs, with 95% confidence intervals obtained through bootstrap resampling to ensure statistical reliability.

Hyperparameter tuning

Table 3 provides details of hyperparameter configurations for our proposed model, systematically exploring key architectural components across multiple dimensions. For the embedding layer, we investigated dimensions ranging from 16 to 64, with 32 selected as the default setting. The Transformer block underwent extensive tuning, examining the number of attention heads (2–4), Transformer layer configurations (2–4 layers), and hidden feed-forward network dimensions (64–256). Similarly, the residual block’s fully connected layers were explored across dimensions of 32–128, allowing flexibility in feature representation. The multilayer perceptron (MLP) layer’s hidden dimension was varied between 64 and 256, with 128 chosen as the default. This methodical approach enables a structured exploration of the model’s architectural hyperparameters, balancing computational efficiency with potential performance gains by systematically varying key network components. Ultimately, the best hyperparameters were selected based on the evaluated performance of the model on the validation set.

The proposed model and all baseline models were trained for up to 50 epochs using the Adam optimizer (Kingma & Ba, 2014) with a learning rate of 0.01. This learning rate was determined through a grid search over the following values: 0.001, 0.005, 0.01, 0.05, with validation performance guiding the selection. A batch size of 128 was used for all experiments. To prevent overfitting, dropout was applied with a rate of 0.3, and L2 weight regularization was also utilized. Early stopping based on validation, AUPRC was implemented with a patience of 10 epochs. Each experiment was repeated five times with different random seeds to assess consistency, and the best-performing model on the validation set was chosen for testing. All model parameters, architectural configurations, and tuning strategies were kept consistent across all comparisons to ensure a fair evaluation.

Comparison with baseline models

The performance of our proposed model was evaluated against several established ML and DL models across eight well types. The comparative analysis incorporated multiple ML models: SVM (Alvarsson et al., 2016) for their optimal hyperplane classification capabilities; Gaussian process classifier (GPC) Villacampa-Calvo et al. (2020) utilizing probabilistic function distributions; and Random Forest classifier (RFC) (Breiman, 2001) leveraging ensemble-based decision tree integration. Tree-based methods were further represented by the eXtreme Gradient Boosting (XGB) (Chen & Guestrin, 2016; Deng et al., 2021) model, which implements sequential tree construction with error correction mechanisms, and the fundamental decision tree (DT) (Wu et al., 2007) model based on feature-driven decision paths. The instance-based $k$-nearest neighbors ( $k$-NN) (Mucherino, Papajorgji & Pardalos, 2009) model was included for its proximity-based classification methodology. For DL approaches, we used the artificial neural network (ANN) (McCulloch & Pitts, 1943), characterized by its neural layered architecture, and the CNN (O’Shea & Nash, 2015), distinguished by its hierarchical feature learning through convolution operations.

Comparison with deep learning models

To conduct a comprehensive performance analysis, we also compared our proposed model with several advanced DL methods that have been implemented on tabular data. These methods include DCN-V2 (Wang et al., 2020), an architecture for learning-to-rank that uses cross-network and self-network components to efficiently capture feature interactions, improving upon the original deep and cross network (DCN) model. TabNet (Arik & Pfister, 2019) is a deep tabular data learning architecture that applies sequential attention to choose which features to reason from at each decision step, enabling interpretability. FT-Transformer (Feature Tokenizer + Transformer) is a simple adaptation of the Transformer architecture, originally designed for language modeling, to the tabular domain by tokenizing features (Gorishniy et al., 2021). TabNN (Ke et al., 2019) is a universal neural network solution that automatically derives effective architectures customized to specific tabular datasets and predictive modeling tasks.

Comparison with baseline models

Table 4 compares the performance of the proposed model against several baseline ML and DL models across eight well types. It is evident that the proposed model consistently outperforms the other models in terms of both AUROC and AUPRC across all well types. The proposed model achieves the highest AUROC and AUPRC values for each well type, demonstrating its superior classification performance compared to the baseline models. Among the baseline models, XGB and $k$-NN generally perform better than the other ML models, with XGB showing the highest AUROC and AUPRC values for most well types. The DL models, ANN and CNN, also demonstrate competitive performance, often surpassing the ML models but falling short of the proposed model’s performance.

The performance gap between the proposed model and the baseline models varies across well types. In the SHRIMPLIN well type, the proposed model achieves an AUROC of 0.908 and an AUPRC of 0.580, significantly outperforming the best baseline model, XGB, which has an AUROC of 0.870 and an AUPRC of 0.450. In contrast, for the NOLAN well type, the performance gap is smaller, with the proposed model achieving an AUROC of 0.864 and an AUPRC of 0.418, while the best baseline model, XGB, has an AUROC of 0.844 and an AUPRC of 0.406. Overall, the results presented in the table demonstrate the effectiveness of the proposed model in classifying various well types, showcasing its superior performance compared to established ML and DL baseline models.

Comparison with deep learning models

The experimental results demonstrate compelling evidence of our model’s superior performance across multiple well types and evaluation metrics compared to existing state-of-the-art approaches (Table 5). In terms of AUROC values, our model consistently outperforms all competitors, with particularly impressive results in LUKA GU (0.924) and SHRIMPLIN (0.908) wells, representing substantial improvements over the second-best performer, the FT-Transformer. The AUPRC values further underscore our model’s capabilities, maintaining significant leads across all well types, notably in SHRIMPLIN (0.580), LUKA GU (0.575), and NEWBY (0.552) wells. Among competing models, the FT-Transformer emerges as the consistent second-best performer, particularly in AUROC values, while the baseline DCN model shows the weakest performance across all categories. The consistent performance hierarchy across different well types suggests that our model’s superior results represent a fundamental improvement in the underlying modeling approach rather than being limited to specific well characteristics. These results strongly indicate that our proposed architecture successfully addresses key limitations of existing approaches, particularly in handling complex well data characteristics and class imbalance issues.

Modeling reproducibility

The modeling reproducibility analysis of our proposed method demonstrates robust and consistent performance across multiple trials (Table 6). Mean, standard deviation (SD), and 95% confidence intervals (95% CI) were computed investigate the performance variation of the method. The model achieved an AUROC value of 0.883 $\pm$ 0.027 and an AUPRC of 0.502 ± 0.071, with 95% CI of [0.863–0.903] and [0.449–0.555], respectively. Notable performance peaks were observed in trial 5, which recorded the highest AUROC of 0.924 and AUPRC of 0.575, while trial six showed relatively lower metrics with an AUROC of 0.841 and AUPRC of 0.407. The narrow SD in AUROC value suggests stable model performance across different trials, though the slightly higher variability in AUPRC values indicates some sensitivity to class imbalance in the dataset. The small CI for both metrics further support the reliability of the model’s performance, providing strong statistical evidence for the reproducibility of our methodology.

Limitations

Despite promising outcomes, our method has shortcomings that need to be addressed in the future. The model’s reliance on high-quality labeled training data may limit its applicability in regions with sparse or uncertain facies annotations. While the dual-stream architecture enhances feature processing, it introduces additional computational overhead compared to simpler approaches, potentially affecting deployment in systems with limited computing resources. The current implementation assumes consistent well-log measurement types across all wells, which may not hold in cases involving legacy data or varying logging tool specifications. The model’s performance shows some sensitivity to class imbalance, as evidenced by the variability in AUPRC values across different facies types. In terms of applicability, the proposed method demonstrates promising performance in facies classification, but its practical implementation is constrained by requirements for high-quality labeled training data, consistent well-log measurement types, and computational resources.