A semi-permeable attention network for ESG score prediction

Semi-permeable attention

Let a tabular instance be represented by F feature tokens with queries $Q\in {\mathbb{R}}^{F\times d}$, keys $K\in {\mathbb{R}}^{F\times d}$ and values $V\in {\mathbb{R}}^{F\times d_v}$. The (unmasked) self-attention logits and weights are

(1) $$L=\frac{Q{K}^{\mathrm{\top }}}{\sqrt{d}}\in {\mathbb{R}}^{F\times F},{\alpha }_{ij}=\frac{\exp (L_{ij})}{{\sum }_{k=1}^F\exp (L_{ik})},\mathrm{A}\mathrm{t}\mathrm{t}\mathrm{n}(Q,K,V)=\alpha V.$$Throughout, we define information flow as key $\to$query, because the output for query $i$ aggregates values from keys $j$ via ${\sum }_j{\alpha }_{ij}{V}_j$.

We wish to let weaker (less informative) features borrow signal from stronger ones, while protecting strong features from contamination by weak ones. Equivalently, a query $i$ may attend only to keys $j$ that are at least as informative as itself.

Let $s\in {\mathbb{R}}^{\mathbb{F}}$ denote feature-importance scores (larger = more informative). Semi-Permeable Attention (SPA) injects an asymmetric mask $M\in {\mathbb{R}}^{F\times F}$ into the logits before the softmax:

(2) $${\alpha }_{ij}=\frac{\exp (L_{ij}+M_{ij})}{{\sum }_{k=1}^F\exp (L_{ik}+M_{ik})}.$$To implement the intended policy (high $\to$low allowed; low $\to$high blocked in the key $\to$query sense), we set

(3) $$M_{ij}=\{\begin{array}{cc}0,\hfill & s_j\ge s_i,\hfill \\ -\gamma ,\hfill & s_j<s_i,\hfill \end{array}\gamma \gg 0.$$Thus, if query $i$ is weak and key $j$ is strong ( $s_j>s_i$), the edge is allowed ( $M_{ij}=0$); if $i$ is strong and $j$ is weak ( $s_j<s_i$), the edge is suppressed ( $M_{ij}=-\gamma$). In practice we use a large finite $\gamma$ (e.g., 10– ${10}^6$) so that $\exp (-\gamma )$ is numerically negligible, avoiding $-\mathrm{\infty }$.

In all experiments, $s$ is computed on the training split only to avoid leakage. Practical choices include mutual information with the target (default), permutation importance (trees), tree-based Gini importance, or SHAP aggregates. We report Kendall’s $\tau$ between rankings to verify robustness.

For settings where a fixed $s$ may be suboptimal, a differentiable soft mask can be used:

(4) $$g_{ij}=\sigma (\beta (s_j-s_i)),{\tilde{M}}_{ij}=\log g_{ij},{\alpha }_{ij}=\frac{\exp (L_{ij}+{\tilde{M}}_{ij})}{{\sum }_k\exp (L_{ik}+{\tilde{M}}_{ik})},$$where either $s$ is learned directly or produced by a small scoring network from feature embeddings. The temperature $\beta$ can be annealed upward so $g_{ij}$ approaches a step function; mild regularization (e.g., entropy on $g$ or ${\ell }_1$ on deviations from an initializer $s_0$) prevents degeneracy.

SPA adds a broadcastable $F\times F$ mask per head; the overhead is negligible relative to the $O(F^{2}d)$ attention. For completeness, a “reverse” policy (the opposite inequality) can be defined when one prefers strong queries to aggregate from weak keys; all results in this article use the protect-strong policy above.

Feature preprocessing

The Feature Preprocessing module transforms raw tabular data into a uniform format that the neural network can process. The system handles mixed data types through two operations: normalization of numerical features and encoding of categorical features. Numerical features are normalized using standard scaling as:

(5) $$f_{num}^{\prime }=\frac{f_{num}-\mu }{\sigma },$$where $\mu$ is the mean and $\sigma$ is the standard deviation. The normalization helps to prevent large-value features from overwhelming smaller ones, while categorical features are converted to meaningful numbers using CatBoost Encoding as:

(6) $$f_{cat}^{\prime }=\mathrm{C}\mathrm{a}\mathrm{t}\mathrm{B}\mathrm{o}\mathrm{o}\mathrm{s}\mathrm{t}\mathrm{E}\mathrm{n}\mathrm{c}\mathrm{o}\mathrm{d}\mathrm{e}\mathrm{r}(f_{cat}).$$

The standardized features then flow into the Embedding layer to create rich representations using:

(7) $$z_i=\tanh (f_i^{\prime }{W}_{i,1}+b_{i,1})\odot (f_i^{\prime }{W}_{i,2}+b_{i,2}),$$where $f_i^{\prime }$ is the pre-processed feature, $W_{i,1},W_{i,2}$ are learnable weights, $b_{i,1},b_{i,2}$ are biases, and $\odot$ represents element-wise multiplication. This embedding transformation converts simple preprocessed values into complex, multi-dimensional representations that capture deeper patterns in the data.

Prediction head

The Prediction Head module is the final decision-making component, transforming the rich feature representations learned by the transformer architecture into actionable predictions through a carefully designed two-layer structure. This module employs sequential processing where the first fully connected layer (Linear_f) compresses information along the feature dimension, followed by a second layer (Linear_d) that integrates information across the embedding dimension, creating a comprehensive fusion of all learned patterns. The mathematical operation follows the formula:

(8) $$p=\varphi \left({\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}_d\left(\mathrm{P}\text{-}\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}\left({\mathrm{L}\mathrm{i}\mathrm{n}\mathrm{e}\mathrm{a}\mathrm{r}}_f\left(z^{(L)}\right)\right)\right)\right),$$where $z^{(L)}$ represents the processed output from the model’s final Gated Linear Unit (GLU) module, P-ReLU provides non-linear activation between the layers, and $\varphi$ serves as the task-appropriate final activation function. The versatility of this design allows it to adapt seamlessly to different ML tasks. For multi-class classification problems, it outputs C values (where C equals the number of target categories) using softmax activation to generate probability distributions, while for binary classification and regression tasks, it produces a single output value using sigmoid activation.

Rationale for method selection

The choice of modeling techniques in this study was driven by the unique challenges posed by ESG data: its tabular format, limited sample size, and varying feature importance across dimensions. Traditional deep learning architectures, while effective in image or text domains, often underperform on structured tabular datasets due to their assumption of uniform feature interaction and high data requirements. To address this, we adopted the ExcelFormer architecture, which is specifically designed for tabular data and has demonstrated strong generalization performance across diverse tabular prediction tasks.

Within this architecture, the SPA mechanism plays a central role. Unlike conventional self-attention that treats all features equivalently, SPA introduces an asymmetric information flow that allows more informative features to selectively receive signals from less informative ones, while preventing the reverse. This design aligns well with the nature of ESG data, where certain features (e.g., governance indicators) may carry greater predictive value than others depending on the task. Additionally, the ExcelFormer architecture avoids the need for extensive hyperparameter tuning, making it suitable for practical deployment in scenarios with limited domain-specific modeling expertise.

The overall model architecture, incorporating feature importance-aware attention, rich embedding layers, and a task-adaptive prediction head, was thus selected to maximize robustness, interpretability, and predictive accuracy in ESG forecasting. This deliberate alignment between data characteristics and modeling strategy forms the foundation of the proposed method.

It should be noted that, beyond credit risk, recent advances across adjacent domains show that transformer representations and robust ensemble strategies can materially improve predictive performance on structured problems (Nguyen-Vo et al., 2021; Nguyen et al., 2022). In parallel, large-scale mathematical modelling has successfully supported high-stakes policy decisions, underscoring the value of transparent, data-driven decision support frameworks (Nguyen et al., 2021). Also, to assess robustness under limited or imbalanced covariate coverage, practitioners may optionally augment training data with modern tabular generators. In particular, transformer-based VAEs for tables and Wasserstein-regularised autoencoders can produce high-fidelity synthetic rows while preserving marginal/conditional structure (Wang & Nguyen, 2025a, 2025b). However, in this work, we do not rely on synthetic data for our main results.

Proposed model

Figure 2 visualizes the model architecture proposed in the study. The model’s learning principle centers around a SPA mechanism that addresses the unique challenges of tabular data processing. The architecture begins with an importance-weighted feature embedding stage where raw tabular features $(x_1,x_2,x_3,x_4)$ are first processed through the Feature Preprocessing module to compute Importance (I), which represents feature-specific weights. These weights are applied before transforming the input from raw features $\mathbf{x}\in {\mathbb{R}}^{\mathbb{F}}$ to embeddings $\mathbf{z}\in {\mathbb{R}}^{F\times d}$. This importance weighting recognizes that tabular features have varying significance, unlike image pixels or text tokens that are typically treated more uniformly. The SPA mechanism uses three linear layers to create three vectors: Query (Q), Key (K), and Value (V). The selective information flow is designed to prevent inappropriate feature mixing—a critical issue in heterogeneous tabular data. After the attention module, the output is fed into GLU layers, with $\tanh$ activation introducing necessary non-linearity. The architecture also incorporates a masking strategy to mask certain features during training to force the model to learn robust feature representations and inter-feature dependencies. Finally, the GLU’s output ${\mathbf{z}}^{\prime }\in {\mathbb{R}}^{F\times d}$ is passed to the Prediction Head module to produce the final predicted values (Algorithm 1).

Evaluation metrics

Model performance was evaluated using several regression metrics including Root Mean Squared Error (RMSE), Mean Absolute Error (MAE), Mean Squared Error (MSE), and the coefficient of determination (R²). The mathematical formulations for these metrics are defined as follows:

(9) $$\mathrm{M}\mathrm{S}\mathrm{E}=\frac{1}{n}\sum _{i=1}^n{(y_i-{\hat{y}}_i)}^2,$$

(10) $$\mathrm{R}\mathrm{M}\mathrm{S}\mathrm{E}=\sqrt{\frac{1}{n}\sum _{i=1}^n(y_i-{\hat{y}}_i{)}^2},$$

(11) $$\mathrm{M}\mathrm{A}\mathrm{E}=\frac{1}{n}\sum _{i=1}^n|y_i-{\hat{y}}_i|,$$

(12) $${\mathrm{R}}^2=1-\frac{{\sum }_{i=1}^n{(y_i-{\hat{y}}_i)}^2}{{\sum }_{i=1}^n{({\hat{y}}_i-\overline{y})}^2},$$where $y_i$ represents the actual values, ${\hat{y}}_i$ denotes the predicted values, $\overline{y}$ is the mean of actual values, and $n$ is the total number of observations.

Comparison with other methods

To ensure a fair and comprehensive evaluation, we compared our proposed model with both traditional ML methods and DL approaches tailored for tabular data. The ML baselines included five commonly used algorithms: Random Forest (RF), Decision Tree (DT), AdaBoost, and XGBoost (XGB), alongside our proposed method. In addition, we benchmarked our model against four state-of-the-art DL architectures designed for structured data, including FT-Transformer, TabTransformer, TabFPN, and ARM-Net. All models were reimplemented under consistent experimental settings to facilitate an objective comparison across three ESG prediction tasks. This comparative framework allows us to assess the relative strengths of our approach in the context of both conventional and modern methodologies.

Model training and optimization

The model was trained for 100 epochs and optimized using the Adam optimizer with a learning rate of $5\times {10}^{-4}$. The MSE was used as the loss function during training. The entire modeling pipeline was implemented using the PyTorch 1.13 framework and executed on an NVIDIA RTX 4,080 graphics card equipped with 16 GB of VRAM. Data preprocessing and computational workflows were conducted in an Ubuntu 22.04 LTS environment running on an AMD Ryzen 7 5800X processor (3.8 GHz base frequency, 4.7 GHz boost clock) supported by 64 GB of system memory.

Ablation study

The ablation study was conducted to independently evaluate the contribution of each key component in our proposed model across three critical design choices (Table 2). To investigate the effect of feature encoding, we compared our model with two other variants in which CatBoost was replaced by one-hot and target-mean encoding. The results reveal that CatBoost encoding consistently outperforms both one-hot and target-mean encoding across all ESG prediction tasks. To analyze the effect of SPA, we compared our model with its variant without SPA applied. The inclusion of SPA shows substantial improvements over the baseline without SPA, with notable reductions in MSE up to 17% and corresponding increases in R² scores across all tasks. Finally, we compared our model which uses hard masking with its variant using soft masking. The results show that the hard masking approach slightly outperforms the soft mask strategy, particularly evident in the Environmental and Overall ESG tasks where hard masking achieves better generalization. These results collectively demonstrate that each proposed component contributes meaningfully to the model’s performance, with the combination of CatBoost encoding, SPA mechanism, and hard masking yielding optimal results across diverse ESG prediction scenarios.

Comparison with machine learning baselines

When comparing with ML baselines, our proposed method consistently demonstrates higher performance across all evaluation metrics and ESG prediction tasks (Table 3). In the MSE metric, the proposed method achieves the lowest error rates ranging from 0.0043 for Environmental tasks to 0.0233 for Governance tasks. Conversely, the DT model exhibits the poorest performance across all metrics, with MSE values substantially higher than other methods across all tasks. The proposed method has the highest performance, followed by XGBoost, AdaBoost, RF, and DT as observed across different metrics. Prediction of the Environmental scores emerges as the most predictable task across all models, achieving R² values above 0.85, with the proposed method reaching 0.94. In contrast, the prediction of Social and Governance scores present greater challenges, with R² values ranging from about 0.41 to 0.64, respectively, indicating substantially lower predictive accuracy. The overall ESG scores reflect the averaged performance across all three tasks. The proposed method shows consistent improvements over traditional ML methods, with lower error rates compared to XGBoost and more substantial improvements over RF across most tasks. The consistency of these improvements across all four evaluation metrics confirms the robustness of the proposed approach for ESG prediction tasks. Fine-tuned values of the ML models are provided in Table 4.

Comparison with deep learning baselines

Besides ML methods, the results in Table 5 show that our proposed method outperforms all competing DL models across every metric. It maintains MSE values between 0.0043 (Environmental) and 0.0233 (Governance), representing the lowest error rates in the comparison. TabTransformer delivers the poorest results, consistently showing higher error values and lower R² scores across all ESG tasks. Our model ranks first, followed by ARM-Net, FT-Transformer, and TabTransformer. Environmental score prediction performs well by all DL models, with R² values above 0.91 and our method reaching 0.94. Both Social and Governance predictions pose more significant challenges, achieving R² ranges of 0.54–0.60 and 0.56–0.64 respectively. Our approach shows meaningful improvements over ARM-Net, the second-best performer, while achieving more substantial gains compared to transformer-based architectures like FT-Transformer and TabTransformer. These consistent performance advantages across all metrics demonstrate the robustness of our DL framework for ESG prediction. Information on modeling details of all DL models are summarized in Table 6.

Limitations

Despite achieving superior performance across all evaluation metrics, our proposed method inherits several limitations from the ExcelFormer architecture. The approach demands higher computational resources and extended training periods compared to traditional ML methods, potentially limiting deployment in resource-constrained environments. While our implementation incorporates architectural regularization through the Semi-Permeable Attention mechanism and gated linear units, the current regularization strategies may not be sufficient for diverse tabular datasets with varying characteristics, potentially limiting generalization performance in data-scarce scenarios. Nevertheless, our experimental validation demonstrates that these limitations are outweighed by the consistent performance improvements our method achieves over both baseline and advanced approaches across all ESG prediction tasks.