Quantification of left ventricular mass in multiple views of echocardiograms using model-agnostic meta learning in a few-shot setting

Data preprocessing

The input images were resized to 256 × 256 pixels, and augmentation was applied with rotations ranging from 0 to 30 degrees. Additionally, the images were normalized to a range of 0 to 255 pixels, and the batch size was set to k (5, 10, 20, and 30 shots).

Annotations were conducted by both a cardiologist and an echocardiographer using the LabelMe program (Torralba, Russell & Yuen, 2010). All labeled data were double-checked by inter-observers, and in cases of disagreement during the validation process, annotations were revised through discussion.

Dataset preparation

In this study, we conducted experiments on four different views of echocardiography: A2C, A4C, PLAX, and PSAX, using several open echocardiography datasets. Specifically, we utilized the publicly available EchoNet-LVH dataset (Ouyang et al., 2019) for the PLAX view and the TMED-2 dataset (Huang et al., 2022) for the PSAX view. Additionally, we utilized the publicly available CAMUS dataset (Leclerc et al., 2019) for both the A2C and A4C views.

The training set consisted of 100 baseline images, while the MAML method was composed of k-shot subsets with 5, 10, 20, and 30 images (Fig. A2). The validation set comprised 34 images for the A2C view, 30 images for the A4C view, 100 images for the PLAX view, and 30 images for the PSAX view. The datasets used were randomly sampled from each view’s public dataset, ensuring no overlap between patients, and the test set comprised 100 images for each view.

Semantic segmentation models

The experimented segmentation models were U-Net (Ronneberger, Fischer & Brox, 2015), and DeepLab-v3+ (Chen et al., 2018). Firstly, U-Net consists of a contracting path (encoder) to capture context and a symmetric expanding path (decoder) to enable precise localization. The architecture uses skip connections between corresponding layers in the encoder and decoder. These connections help the network retain spatial information lost during down-sampling, improving segmentation accuracy, especially for tasks that require detailed boundary predictions. In this study, the Efficientnet-B0 (Tan & Le, 2019) was used as the encoder model.

Additionally, DeepLab-v3 builds upon earlier versions by incorporating atrous (dilated) convolutions, which help capture multi-scale contextual information without increasing the computational cost. Atrous convolutions allow the network to enlarge the receptive field without losing resolution. DeepLab-v3 also uses a technique called the Atrous Spatial Pyramid Pooling (ASPP) module, which applies different rates of dilation to capture information at multiple scales. This architecture is particularly effective at segmenting objects in images with complex backgrounds.

Heatmap-based point estimation in echocardiography

The segmentation models utilized the heatmap-based point estimation method (Bulat & Tzimiropoulos, 2016; Zha et al., 2023) to quantify LVM for predicting specific four points within echocardiograms. Echocardiographic images suffer from low contrast and poorly defined chamber borders, making precise landmark regression unreliable (Dudnikov, Quinton & Alphonse, 2021; Mogra, 2013). To overcome this, we convert each target point into a small Gaussian-shaped region and treat detection as a segmentation task rather than direct coordinate regression. Concretely, for each ground-truth landmark we generate a heatmap by placing a 2D Gaussian kernel—centered at the true point and with standard deviation σ—over the surrounding pixels.

The cardiologist and the echocardiographer initially annotated the desired locations as point coordinates. From these points, a two-dimensional gaussian distribution (mask or contour) with a deviation of 7 was automatically generated and used as labels (Fig. A3). As a result, the segmentation model trained on this dataset generates a heatmap resembling a gaussian distribution. The model calculates the center points of the identified regions, ultimately determining specific point coordinates within the echocardiogram.

Model-agnostic meta learning

MAML (Finn, Abbeel & Levine, 2017) is a method of meta-learning designed to learn a good set of initial parameters through various processes, enabling rapid adaptation to new tasks. MAML is not constrained by the model structure and can generally be applied to a variety of learning problems, thus delivering strong performance in tasks that require rapid adaptation with little data and only a few examples.

MAML involves a two-stage optimization process (Finn, Abbeel & Levine, 2017). The inner optimization starts with initial model parameters and adjusts these parameters through a few learning steps for rapid adaptation to individual tasks. The outer optimization integrates learning outcomes from various tasks to update the initial model parameters. This study applied three prominent MAML methods—first-order model-agnostic meta learning (FOMAML) (Finn, Abbeel & Levine, 2017), Meta-SGD (Li et al., 2017), and meta-curvature (Park & Oliva, 2019)—to multiple views of echocardiograms.

The reason for comparing these segmentation models using MAML-based few-shot learning methods is to evaluate their adaptability and performance in scenarios where data is scarce. Few-shot learning is particularly valuable in real-world applications where gathering large annotated datasets for segmentation tasks can be difficult. By using these MAML methods, the goal is to identify which variant can achieve better model generalization with fewer samples. Each MAML variant has unique properties regarding computational efficiency and adaptability, and this comparison helps pinpoint the optimal trade-offs between performance and resource usage when combined with the segmentation models.

First-order model-agnostic meta learning

First-order model-agnostic meta learning (FOMAML) (Finn, Abbeel & Levine, 2017) simplifies the computational complexity in MAML by omitting the second derivative calculations that assess the impact of parameter updates on the overall model parameters after each task. This omission significantly reduces computational demands while maintaining similar performance. FOMAML approximates second-order derivatives with first-order derivatives and updates the meta-parameters with a few gradient descent steps. This makes meta-learning more practical and scalable for few-shot learning and related tasks. At the k-th inner gradient iteration, the model parameter is ${\theta }_k$, the learning rate $\alpha$, and the loss $\mathcal{L}$, as shown in Eq. (1) (Finn, Abbeel & Levine, 2017).

(1) $${\theta }_k={\theta }_{k-1}-\alpha {\mathrm{\nabla }}_{\theta }{\mathcal{L}}^{\left(0\right)}\left({\theta }_{k-1}\right).$$

During the outer iteration, the meta-objective was updated for each batch. FOMAML (Finn, Abbeel & Levine, 2017) simplifies the equation by omitting the second derivative term of MAML, and the initial model parameter ${\mathrm{\theta }}_0$ is ${\mathrm{\theta }}_{\mathrm{m}\mathrm{e}\mathrm{t}\mathrm{a}}$, defined as shown in Eqs. (2), (3) (Finn, Abbeel & Levine, 2017)_.

(2) $${\theta }_{meta}={\theta }_{meta}-\beta g_{FOMAML}$$

(3) $$g_{FOMAML}={\mathrm{\nabla }}_{{\theta }_k}{\mathcal{L}}^{\left(1\right)}\left({\theta }_k\right).$$

Meta-SGD

Meta-SGD (Li et al., 2017) enhances the flexibility and efficiency of meta-learning by adding the capability to automatically adjust learning rates through the meta-learning process. While traditional MAML methods apply the same learning rate to all parameters, Meta-SGD (Li et al., 2017) learns individual learning rates for each model parameter. This allows for the adjustment of learning speeds according to the importance of each parameter during the meta-learning process. Meta-SGD (Li et al., 2017) involves initializing and adapting the learner $f_{{\theta }_i}$ in the meta-space ( $\theta ,\alpha$), where the learning rate $\alpha$ is typically set manually and the learner is updated through iterative gradient descent starting from a random initialization as shown in Eq. (4) (Li et al., 2017).

(4) $${\theta }_i={\theta }_{i-1}-{\alpha }_{i-1}\cdot {\mathrm{\nabla }}_{\theta }{\mathrm{\Sigma }}_{{\mathcal{T}}_i}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{\theta }_{i-1}}\right)$$

In the inner iteration, weights are updated while in the outer iteration, both the meta-parameters ${\theta }_k$ and ${\alpha }_k$ are updated. The learning process is conducted with a learning rate of $\beta$ for the task distribution $p\left({\mathcal{T}}_i\right)$, which defined as shown in Eqs. (5), (6) (Li et al., 2017).

(5) $${\theta }_k={\theta }_{k-1}-\beta {\mathrm{\nabla }}_{\theta }{\mathrm{\Sigma }}_{{\mathcal{T}}_i\sim p\left({\mathcal{T}}_i\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{\theta }_i}\right)$$

(6) $${\alpha }_k={\alpha }_{k-1}-\beta {\mathrm{\nabla }}_{\alpha }{\mathrm{\Sigma }}_{{\mathcal{T}}_i\sim p\left(\mathcal{T}\right)}{\mathcal{L}}_{{\mathcal{T}}_i}\left(f_{{\theta }_i}\right)$$

Meta-curvature

Meta-curvature (Park & Oliva, 2019) retains the basic meta-learning structure used in MAML but adds curvature information to each parameter’s update, allowing for more appropriate updates for each task-specific parameter. This curvature information reflects how parameter updates are influenced by the shape of the task’s loss function, enabling more sophisticated adjustments than the typical learning rates or parameter update methods. The key concept of meta-curvature (Park & Oliva, 2019) is to adjust the optimization path to suit the characteristics of each task. It achieves this by learning a meta-trained matrix used in parameter updates, which reflects the unique characteristics of each task’s loss surface. The meta-trained matrix $M_{mc}$ can be used to expand all the same-sized meta-curvature matrices ${\hat{M}}_o$, ${\hat{M}}_i$ and ${\hat{M}}_f$ with the Kronecker product $\otimes$, and a k-dimensional identity matrix $I_k$, which defined as shown in Eqs. (7)–(9) (Park & Oliva, 2019).

(7) $${\theta }_i={\theta }_{i-1}-M_{mc}\cdot {\mathrm{\nabla }}_{\theta }{\mathrm{\Sigma }}_{T_i}{\mathcal{L}}_{T_i}\left(f_{{\theta }_{i-1}}\right)$$

(8) $$M_{mc}={\hat{M}}_o{\hat{M}}_i{\hat{M}}_f$$

$${\hat{M}}_o=M_o\otimes I_{C_{in}}\otimes I_d,$$

(9) $${\hat{M}}_i=I_{C_{out}}\otimes M_i\otimes I_d,$$

$${\hat{M}}_f=I_{C_{out}}\otimes I_{C_{in}}\otimes M_f.$$

This matrix replaces the traditional scalar learning rate and allows for finer control over the direction and magnitude of parameter updates. This method enhances the adaptability and effectiveness of the learning process by aligning updates more closely with the specific needs of each task (Park & Oliva, 2019).

Almost-no-inner-loop

Almost-no-inner-loop (ANIL) (Raghu et al., 2019) simplifies the standard MAML framework by restricting inner-loop adaptation to only the task-specific “head” parameters, while leaving the shared feature extractor fixed during task-level updates. Concretely, we partition the model parameters into a shared representation $\theta$ and a per-task head $\varphi$, and perform K inner-gradient steps only on $\varphi$:

(10) $${\varphi }_i={\varphi }_{i-1}-\alpha {\mathrm{\nabla }}_{\varphi }{\mathcal{L}}_{T_i}\left(f_{\theta ,{\varphi }_{i-1}}\right)$$with $\theta$ held constant. After K steps, the outer-loop meta-update then adjusts both $\theta$ and $\varphi$ by aggregating task losses evaluated at the adapted head ${\varphi }_k$:

$$\theta \leftarrow \theta -\beta {\mathrm{\Sigma }}_{T_I}{\mathrm{\nabla }}_{\theta }{L}_{T_I}\left(f_{\theta ,{\varphi }_k}\right),$$

(11) $$\varphi \leftarrow \varphi -\beta {\mathrm{\Sigma }}_{T_I}{\mathrm{\nabla }}_{\varphi }{L}_{T_I}\left(f_{\theta ,{\varphi }_k}\right).$$

By eliminating inner-loop updates on $\theta$, ANIL achieves a dramatic reduction in per-task computation and highlights that rapid adaptation primarily occurs in the final layer, while the shared representation learned $\theta$ generalizes across tasks. An illustration of the differences between the MAML and ANIL methods is shown in Supplemental Fig. A4.

Implementation details

The experimented segmentation models were trained using the mean square error (MSE) loss function with the Adam optimizer, employing a learning rate of 5e−3. During the meta-training process, 50 epochs were applied, and the learning rate between adaptations in meta-training was set to 0.03. Adaptation was performed 10 times for all shots, and the learning rate between meta-tests was set to 0.05, with 100 adaptation steps. Additionally, the final model was selected based on the highest mean pixel accuracy observed in the validation set. In the inference process, the highest probability value in the region was extracted as a point and used as the final predicted value. To compare the performance, the baseline was set to 100-shot, and the MAML methods were varied by 5, 10, 20, and 30 shots.

The development was implemented using PyTorch (ver. 1.12.1) and learn2learn (ver. 0.1.7), and the model was trained using NVIDIA RTX A6000 GPU on Linux (Ubuntu 18.04). Our implementation is available here: https://github.com/KimYeongHyeon/Metalearning-echocardiogram. Here, the pre-trained model weights are also made available, along with dataset selected and labeled from the raw public data. Specifically, EchoNet-LVH dataset (https://echonet.github.io/lvh/index.html#dataset) (Ouyang et al., 2019) is used for the PLAX view, the TMED-2 dataset (https://tmed.cs.tufts.edu/tmed_v2.html) (Huang et al., 2022) for the PSAX view, and the CAMUS dataset (https://www.creatis.insa-lyon.fr/Challenge/camus/databases.html) (Leclerc et al., 2019) for both the A2C and A4C views. These resources are provided for the MAML experiments in this study.

Evaluation metrics

The proposed method was quantitatively evaluated using the mean distance error (MDE), which calculates the difference between the reference points and the coordinates predicted by the model. Additionally, we introduced a new evaluation metric, successful detection rate (SDR), which evaluates whether predicted points fall within a certain threshold distance. Furthermore, the model predicted line segments formed by the points such as the intraventricular septum (IVS), left ventricular internal dimension (LVID), and left ventricular posterior wall (LVPW) in the PLAX view (Kristensen et al., 2022). The mean angle error (MAE), which measures the angle difference between the predicted line and the reference line, was also evaluated.

Furthermore, we introduce a novel evaluation metric, spatial angular similarity (SAS), which jointly captures both the degree of parallelism (angular similarity) and the distance difference between two lines by integrating elements of SDR and MAE. The metric is defined as:

(12) $$SAS=100\times (\alpha \cdot \left|cos\theta \right|+\left(1-\alpha \right)\cdot e^{-\beta d}.$$

Here, α (ranging from 0 to 1) weights the orientation component, d denotes the distance between the two lines, β determines the rate at which the distance-based score decays, and perfectly parallel lines are guaranteed a minimum score of α × 100 regardless of their separation.

In clinical practice, small errors in landmark localization can lead to substantial inaccuracies in left ventricular mass, since LVM formulas combine linear measurements (e.g., septal thickness, cavity diameter) in three-dimensional measures. Our suite of metrics therefore provides a multi-faceted assessment of how well the model meets the precision demands of LVM quantification.

Performance comparison of segmentation models

In this study, we utilized the U-Net (Ronneberger, Fischer & Brox, 2015), DeepLab-v3+ (Chen et al., 2018) and SegFormer (Xie et al., 2021) models along with the heatmap-based point estimation method (Bulat & Tzimiropoulos, 2016; Zha et al., 2023) to quantify LVM by predicting specific four points within echocardiograms. We compared the performance of these two models under the baseline setting (100-shot) and used the dice coefficient and mean pixel accuracy as evaluation metrics.

Table 1 presents the results of evaluating four models using the Dice coefficient, mean pixel accuracy, mean distance error and spatial angular similarity metrics. The similarity in performance suggests that the model type does not significantly impact the outcomes, and the lower results from the dice coefficient metric—especially for SegFormer—can be attributed to training with a mean square error loss function on gaussian distribution labels in heatmap form rather than binary masks. Based on the balance of accuracy and localization, DeepLab-v3+ (Chen et al., 2018), which represents superior performance, was selected as the baseline model for subsequence metal-learning experiments.

Performance comparison of MAML methods

We implemented a segmentation model, DeepLab-v3+ (Chen et al., 2018) which demonstrated the best performance of mean pixel accuracy, to locate specific four points within an echocardiogram using several MAML methods, including FOMAML (Finn, Abbeel & Levine, 2017), Meta-SGD (Li et al., 2017), Meta-Curvature (Park & Oliva, 2019) and Almost-No-Inner-Loop (ANIL) (Raghu et al., 2019). The quantitative evaluation of the MAML methods is presented in Table 2 and Supplemental Tables A1–A6, showing the results of mean distance error (MDE), mean angle error (MAE) and SAS in a few-shot environment from multiple views of echocardiogram. It compares the performance of MAML methods to that of a baseline (100-shot) trained with more labels. As the k-shot increased, the errors of the MAML methods decreased, and when the k-shot reached 30, they exhibited similar or better performance compared to the baseline (p > 0.05). Specifically, in the A2C, A4C, and PLAX views, FOMAML (Finn, Abbeel & Levine, 2017) demonstrated the highest performance; however, there is no statistical difference in performance (p > 0.05). For the PSAX view, the proposed FOMAML (Finn, Abbeel & Levine, 2017) demonstrated statistically significantly higher performance compared to the baseline in terms of the MAE metric (p = 0.0493). The k-shot was low, the performance differences among the MAML methods, but as it increased, the performance saturated.

Figure 2 illustrates the results of successful detection rate (SDR) as an extended evaluation of MDE. It indicates the prediction rate of points within the echocardiogram with the pixel-level threshold varied. The k-shot was set to 30, as the threshold increased, the performance of the MAML methods showed similar or better performance compared to the baseline.

Figure 3 presents qualitative results comparing the ground truth with the prediction results in the baseline (100-shot) and the FOMAML (Finn, Abbeel & Levine, 2017) method with a k-shot of 30. In the A2C and A4C views, the reference points are represented by red points, while the predicted results are depicted as green square points (Figs. 3A, 3B). Additionally, in the PLAX and PSAX views, the reference lines are represented by red points and lines (IVS, LVID, and LVPW), while the predicted results are shown as green square points and lines.