Benchmarking augmentation to enhance A-line and B-line classification with sparse data

Lung artifacts

During LUS examination, the examiner searches for artifacts, evaluates them, and derives the lung condition. These artifacts result from the varying effect of the tissues on ultrasound waves, i.e., reflection, scattering, refraction, and attenuation. Not all artifacts suggest the presence of disease; the presence of some, however, precludes the occurrence of others due to the essence of their physical properties. Several organs, such as the liver and spleen, are also important to locate when evaluating pathologies such as pleural effusion.

LUS examination starts by placing the probe in the intercostal space between two ribs, and by searching for the pleural line. The pleura cushions the lungs and chest cavity. The pleural line consists of two distinct layers: the parietal pleura is a membrane that covers the surface of the pleural cavity, the visceral pleura is a thin, slippery membrane that covers the surface of the lungs. A small amount of fluid in the pleural cavity eliminates friction and allows sliding of the parietal and the visceral pleura during breathing. This causes the lung ultrasound artifact called lung sliding, a sliding motion of the pleural line in lung ultrasound recordings. The pleura is visible in every ultrasound image (if the depth of the ultrasonography (USG) probe is correctly set) and serves as a basic navigation point. It is the first horizontal line/artifact in the analysis of LUS video by doctors (Finley & Rusch, 2011).

After identifying the pleural line and the evaluation of the presence and abscence of lung sliding, two other LUS artifacts are searched for, A- and B-lines. Mostly, their presence or absence alone does not indicate a specific pathology. It is important to evaluate the combined occurrence, distribution and prevalence of these artifacts.

A-lines represent repeated reflections of the pleura manifested as bright horizontal lines. The pleura (see top green box in Fig. 1) is always visible as the first line closest to the probe. The remaining horizontal lines appear below the pleural line at the same distance from each other as the distance between the probe and the parietal line. Artifacts that resemble A-lines but deviate from the equidistant pattern are considered suspicious and require further investigation. For the detection of A-lines, the depth of visible area should be set properly on the machine, otherwise, for example if excessive subcutaneous fat is present, it may be impossible to evaluate the presence of this artifact (Lichtenstein, 2014). In the case of lung sliding absence, observation of A-lines supports but does not definitely confirm the presence of pneumothorax (Volpicelli et al., 2012). This combination of artifacts may be also seen by patients with lung emphysema without pneumothorax.

B-lines (see blue box in Fig. 1) are vertical artifacts arising from the pleural line extending indefinitely and erasing A-lines. Caused by fluid or cells in the sub-pleural region, the lung tissue creates discrete microscopic 3-dimensional aerated structures surrounded by fluid or cells. Under these circumstances, air/liquid interfaces act as specular reflectors, discrete reverberation foci, thus creating vertical beam-like artifacts (Wang et al., 2017). When evaluating B-lines, it is important to take the number and distribution of this artifact into account. Finding 2 or 3 B-lines in one intercostal space may be physiological. More B-lines call for a broader differential diagnosis, B-line presence excludes pneumothorax (Lichtenstein, 2014, Volpicelli et al., 2012).

Data labeling

For labeling, doctors used a colaborative software, the Computer Vision Annotation Tool (CVAT) (CVAT.ai, 2023). This tool offers a set of features tailored for medical image annotation, allowing for efficient and accurate labeling of anatomical structures and pathological features. The use of CVAT streamlines the annotation workflow, promoting consistency and reliability in labeled datasets in various domains. It also allows USG videos to be exported together with labeling and all metadata in standardized formats for easy data processing.

In the annotation process, doctors delineated the boundaries of the detected artifacts by multi-polygonal markers and indicated their class. Every video in our dataset was recorded at 30 frames per second (FPS). Due to time constraints, doctors labeled only every 10th frame of the video. Each video was labeled by one doctor and the annotations were then reviewed by another expert. In case of discrepancies, the two experts reached an agreement through discussion; in some cases the opinion of a third expert was also considered. The labeling process was followed by data cleaning supervised by an IT specialist.

The experiments described here were focused on classification only. Therefore, we do not work with the labeling polygons. Only the information about the presence or absence of a given sign on the frame is considered.

Data processing and filtering

LUS videos recorded by USG machines contained metadata such as probe settings, gain, measurement points, etc. These metadata did not overlap with the USG image area and were therefore removed from video frames. We implemented a simple preprocessing script that detected the USG area in the videos by discerning it from the black background. Our solution is applicable with USG devices of various brands and with various probe types. The information regarding the type of USG device used for data acquisition was retained to maintain comprehensive metadata integrity.

Next, we conducted video filtering and categorization. If a single frame of a video contained multiple distinct types of signs, the video was removed to reduce ambiguity in the dataset and minimize potential issues during model training. If A-lines or B-lines were present in one or more frames, the entire video was labeled as positive for that artifact; otherwise, it was labeled as negative. Table 1 provides a quantitative breakdown of the number of labeled videos per artifact class and ultrasound probe type.

The dataset analysis revealed notable differences in the occurrence of pathological signs, particularly between A-lines and B-lines, across the labeled frames within each video. On average, a video encompasses approximately 37 labeled frames, with each frame annotated for the presence or absence of specific pathological features.

In our dataset, A-lines occur at a notably higher rate, being present in 67% of the labeled frames. B-lines were identified in 29% of the labeled frames (see Tables 2 and 3, respectively).

Multiple LUS recordings were acquired from every patient in this research. In the medical domain it is important to ensure patient split while training models–videos from one subject should be used for a single purpose only–for training, validation or testing of the model. If images from the same patient (or ultrasound video) appear in both training and test sets, a model may learn patient-specific patterns instead of generalizable features, inflating performance metrics. During LUS examination, doctors record videos from six different locations on the chest. By capturing various regions of the thorax, the morphological overlap of the videos from the same patient is minimized. In this case, splitting videos from a single patient but from different locations is admissible between different subsets.

To address class imbalance and ensure proper evaluation of our experiments, we employed stratified 5-fold cross-validation. The dataset was split into training and test sets such that each fold maintained a uniform distribution of probe types and video classes, allowing for balanced representation across all subsets.

We created binary classification models for the classification of each sign separately (A-lines, B-lines). Negative examples were selected from videos labeled as showing no lines. Valid labeled frames were selected for each video, ensuring a balanced representation across classes. In cases where the positive class contained fewer frames than the negative class across all videos, we applied uniform frame dropping from each “No Lines” video to balance class distribution.

We applied negation sampling: in the A-line classification model, the negative class contained “No Lines” and also B-lines examples, and vice versa. The opposite class was balanced using the same frame-dropping technique employed for the negative class in the previous step.

Models

In our study, we used ResNet backbones. ResNet is a well-established convolutional neural network (CNN) architecture proposed in He et al. (2016). ResNet stands out for its innovative use of residual blocks, composed of two convolutional layers with Rectified Linear Unit (ReLU) activation functions. The main distinction of the ResNet block is the inclusion of a non-parameterized shortcut connection. This shortcut connection efficiently passes the output of the previous block directly to the next block without any modification. This design mitigates the vanishing gradient problem and facilitates the training of deeper networks.

During our experiments, we explored different variants of ResNet architectures, including ResNet-18, ResNet-34, and ResNet-52. The primary difference between these architectures is the number of residual blocks employed. Larger architectures can be used for more complex problems but they also have a higher tendency to overfit. During our primary experiments, overfitting occurred with larger models. To mitigate this risk, we conducted comparative experiments and focused on ResNet-18 because of its lower complexity and higher generalization performance.

Besides ResNet, we included Inception architecture (Szegedy et al., 2015) for evaluation alongside ResNet to broaden our comparative analysis. Inception modules serve as a fundamental framework for addressing scale variation within images, thereby enhancing the network’s ability to capture multi-scale features effectively (McNeely-White, Beveridge & Draper, 2020).

The distinguishing feature of Inception modules is their use of convolutional layers at different sizes in conjunction with max pooling using the same input. Subsequently, the outputs from these operations are concatenated together, enabling the model to extract diverse feature maps while maintaining a relatively shallow depth compared to other architectures. We selected the Inception-v3 architecture for our experiments due to its frequent use in medical image analysis.

The training process adhered to established practices to ensure efficacy of the models. We trained separate models for each task via transfer learning, using models pre-trained on the ImageNet dataset to ensure fast convergence. For each model type, we carried out empirical testing with a base model, on which all training hyperparameters were optimized. Later on, the same values were used in all experiments. The learning rate was set to 0.001, and the input resolution to $500\times 600$ pixels, balancing computational efficiency and image detail. Employing the Adam optimizer and early stopping with a patience of 5 epochs mitigated the risk of overfitting. The Binary Cross-Entropy loss function was chosen for its suitability in binary classification tasks.

All training was done on Nvidia RTX A500 24GiB vRAM cards with CUDA version 12.2.1.020. The source code was implemented and run in Python 3.9 with Pytorch 2.13 and MONAI 1.2. For optimal performance, we used the NGC Pytorch container environment (version 23). Experiment results were logged in wandb.

Preprocessing methods

Images sourced from various origins needed to be rescaled to fixed input dimensions compatible with the chosen CNN. This is commonly done using interpolation (Chansong & Supratid, 2021). All down/upsampling was performed with bicubic interpolation. Bicubic kernels preserve local gradient structure better than bilinear and avoid the aliasing artefacts that nearest-neighbour causes in high-frequency speckle regions (Hashemi, 2019; Hirahara et al., 2021). Ultrasound videos are typically taller than they are wide, a result of the ultrasound probe’s scanning angle or the imaging device’s acquisition characteristics, as illustrated in Fig. 2. Consequently, resizing these images by stretching them horizontally may distort the features.

Although such basic resizing might not affect horizontal features, vertical B-lines could be significantly compromised, with the characteristic comet tail shape widened out of proportion. Experimental findings have revealed that incorporating padding during preprocessing leads to improved training outcomes (Alrasheedi, Zhong & Huang, 2023; Tang, Ortis & Battiato, 2019; Haryanto et al., 2020). Our primary experiments also showcased this problem, with our models unable to detect B-lines without the use of padding during training. This limitation persisted when using a small dataset comprising videos with varying size ratios. Incorporating zero padding, a common padding technique, also did not increase the performance of the model. This is caused by artificially introduced zero-value pixels around smaller images, which do not activate during forward or backward propagation, therefore their associated synaptic weights are not modified (Hashemi, 2019). We therefore used edge padding, where the edge pixels (i.e., top/bottom rows and left/right columns) of the original image are repeated in the respective direction (Alrasheedi, Zhong & Huang, 2023).

The final preprocessing pipeline consisted of standardizing resolution to 500 $\times$ 600 pixels, and converting frames to grayscale. No intensity normalisation was performed. This decision was made to retain the raw signal characteristics as output by each machine. Any global normalisation risked introducing artefactual contrasts. Frames were selected using a stratified sampling strategy to balance data across different ultrasound probes and artifact labels. In addition to standard resizing to 500 $\times$ 600, we also tested edge padding that maintains anatomical and artifact structure, which is particularly useful for vertically oriented features. The output data consisted of binary labels indicating the presence or absence of A-lines or B-lines on each annotated frame, used in a supervised classification setting. These labels were generated from medical expert annotations based on the presence of specific polygon types for each artefact.

Augmentation methods

Deep neural networks, mainly CNNs, achieve high performance in various image recognition tasks, yet they often encounter challenges with generalization when trained on limited datasets, leading to overfitting. This issue becomes particularly pronounced in scenarios where data scarcity prevails (Hussain et al., 2017). Augmentation techniques offer a solution to address overfitting by expanding the dataset with additional data points, thereby providing the model with a more diverse set of examples for learning. This approach directly tackles the overfitting problem by enriching the training data and enhancing the model’s capacity to generalize to previously unseen instances (Buslaev et al., 2020).

Augmentation strategies can be divided into two types: augmentation performed on real data and augmentation using synthetically generated data. This article focuses on augmentation techniques applied to an existing dataset.

Affine transformation involves adjusting the geometric properties of images, such as rotation, translation, scaling, and shearing. This technique introduces variations in the spatial orientation of images. Elastic transformation deforms images using spatial deformation fields. Unlike affine transformations, elastic transformations allow for local shape variations without imposing colinearity and aspect ratio constraints. These transformations mimic realistic image variations, thereby contributing to improved model robustness (Nalepa, Marcinkiewicz & Kawulok, 2019). Erasing transformation selectively removes specific regions of an image, replacing them with fixed intensity values or random noise. This approach improves the model’s ability to withstand occlusions and decreases reliance on spurious correlations in the data (Garcea et al., 2023).

We combine different augmentations from these three categories and create a group of augmentations that we call “Basic”, for a full list of the augmentation methods in this group refer to Table 4. These methods are used standardly in most image augmentation solutions, not just for medical data. Other augmentation methods target different characteristics specific to ultrasound data, and were grouped together accordingly. We compared model performance with each augmentation group for benchmarking. All augmentations were applied exclusively to the training subset of each cross-validation fold, while the validation and test subsets remained untouched and contained only the original, radiologist-labelled frames. In every experiment we used the implementations of these augmentation methods from the MONAI open source framework.

We focused our research on pixel-level transformations, which alter pixel values to adjust image characteristics such as brightness, contrast, saturation, and noise. These transformations are particularly useful in grayscale medical imaging modalities, where color-based augmentation is not applicable. Pixel-level transformations aid in standardizing pixel distributions across different imaging protocols, ensuring the model’s robustness to scanner variations (Goceri, 2023).

Various USG machines produce imagery of varying quality and properties–speckle noise, salt and pepper noise, and Gaussian noise are frequent. The most common type of noise is speckle noise, which can have an impact on contrast and other intensity metrics as well as the key visual elements. Sudden image alterations, such as memory cell failure, synchronization errors during digitalization, or incorrect sensor cell activity, might result in the appearance of salt and pepper noise (Vilimek et al., 2022). Electronic circuit noise or sensor noise can cause Gaussian noise. Usually, noise is removed from USG images during preprocessing (Devakumari & Punithavathi, 2020; Nugroho et al., 2016). In contrast, in the second group of augmentation, we tested noise-introducing techniques to let the neural network learn and extract features on noisy images.

In the third group of augmentation methods, we selected techniques that alter image intensity. This was justified as each machine represented in our dataset produces images with different intensities and intensity scales (Stock et al., 2015). Data augmentation serves to improve generalization and robustness of the model. We applied the random bias field augmentation designed for magnetic resonance imaging.

With all augmentation methods, their parameters were set to low values to maintain the most typical characteristics of the frames while also ensuring more robust model training. The parameter values used and their effect were consulted with medical experts and were aligned with variance that can occur during standard examinations (e.g., for rotation, we used angles that replicate the examiner moving the probe into different angles).

Datasets

In our initial experiment, we set out to test the hypothesis that extending the original dataset with historical data, specifically videos of examinations done with other ultrasound probes, would enhance the model’s generalization ability and overall performance. The results, as shown in Table 5, reveal that the model’s performance on A- and B-line detection tasks varied when trained on the original dataset (consisting of only footages taken with Lumify probe) and the extended dataset, contradicting our initial assumption.

The observed results can be attributed to the nature of A-lines and B-lines in the dataset. A-lines were present in a majority of frames, visible in approximately 67% of the annotated frames, whereas B-lines appeared in only about 29% of labeled frames (see Tables 1, 2, 3). This means that B-lines were particularly underrepresented in the annotated data, leading to significant performance improvements when additional training data was introduced. The lack of difference in the effect on the two different models suggests that the dataset extension had a similar impact on both. The most significant increase in performance was seen in TPR for B-lines, with the value rising from 0.196 to 0.534 for Inception-v3 and from 0.082 to 0.329 for ResNet-18. However, it’s important to note that extending the dataset also increased the standard deviation for both NPV and TPR metrics for both tasks and models. This can be attributed to the higher variance in data, which made model performance more sensitive to train-test splits during cross-validation.

Transformation methods

Tables 6 and 7 show the results of training when different transformation methods were used for A- and B-line detection, respectively. For the former, the effect of using edge padding instead of the standard resize for transformation was inconclusive. The values of metrics we considered were somewhat worse for ResNet-18, although these changes were small, around 0.05. A bigger change was only registered for TPR when using the extended dataset. However, when using the original dataset, the standard deviation got significantly smaller, from 0.226 to 0.097, when using edge padding. There was some improvement in the metrics with the Inception-v3 network. F1-score and negative predictive value increased in all setups, while the true positive rate decreased for the original dataset from 0.824 to 0.728. No significant changes in standard deviation were observed, except for NPV on data from the original dataset, where the standard deviation was more than halved.

For B-line detection, we observed a generally positive effect of using edge padding. Although changes were not high in F1-score and NPV, there was an apparent increase in TPR for ResNet when using only the original dataset, with recall increasing from 0.082 to 0.215. Performance dropped only when training Inception-v3 on the extended dataset, with a decrease of 0.04 in F1-score and a slight decrease in NPV, while TPR increased even in this case. Overall, the results suggest that using edge padding instead of resizing leads to improved performance for B-line detection.

Another indication of improvement is observed in the saliency maps (Fig. 4): with edge padding, activation is clearly concentrated in the regions where B-lines occur and does not migrate to the padded borders. In contrast, resizing does not highlight B-line regions: in the top example it produces a small line-like patch unrelated to the pleural tract, and in the bottom example the activation is diffuse and off-target. For A-lines (Fig. 5), saliency overlays are less focal than for B-lines: activation spreads along and below the pleura and is often diffuse, offering weaker discriminative guidance. This qualitative pattern aligns with the systematic performance drop we observed with padding for A-lines (Table 6). This pattern was observed consistently across folds and devices in our dataset. Representative failures are shown in Fig. 6: the first row illustrates an A-line case where resizing outperforms padding, and the second row shows a B-line case where both preprocessing choices yield poor focus and predictions.

The results were in accordance with our hypothesis that, as B-lines represent vertical features that could be distorted by standard resizing, carefully selected transformation and preprocessing methods can greatly influence final performance. When we juxtapose the results obtained using resizing and edge padding, we see that even for A-line detection, performance remained at least on a comparable level.

Augmentation methods

Our final experiments explored the changes in performance metrics for training models when using various combinations of augmentation methods on both the original and the extended dataset. We expected a high variability depending on what augmentation methods were used, and results presented in Tables 8 through 11 support this presumption.

Considering A-line detection with Inception-v3 (Table 8), we observe a mixed augmentation effect on various metrics. Combining all augmentation methods listed in Table 4 resulted in the best performance when using the original dataset. This constitutes a 10 percent increase in F1-score with the original dataset with resizing (from 0.734 to 0.817), and an increase of 0.03 when using padding. Similar increases can be observed when training on the extended dataset (an increase of 0.04 and 0.07, respectively). The most notable difference was in NPV with all augmentation methods used, the value increasing from 0.685 to 0.834, a 0.15 increase. On the other hand, somewhat surprisingly, other combinations of augmentation methods caused a decrease in the F1-score when using padding, and a negative effect could be seen in the other two metrics with resizing.

A more generally positive effect of augmentation was observed when training on the extended dataset. Values of the selected metrics increased in almost all cases, except for TPR when using resize and only basic augmentation methods; the value was, however, still comparable to when no augmentation was used. The best F1-scores were around 0.8, although this could be achieved with different combinations of augmentation methods. When using resize, the combination with no noises proved to be the best, while with padding, a combination of all considered augmentation methods did. This might be due to padding repeating values from the edges of the images, which disrupts data distribution, which in turn can be mitigated by introducing noise. However, when using resize, this is not such a big issue since simple resizing maintains the noise characteristics of the original frames. This trend could also be observed in original data when leaving out noises, which resulted in results comparable to the best improvement when using resize. However, it is important to note that since extending the dataset did not have a largely positive effect on A-line detection, data augmentation did not increase relevant metrics significantly compared to the benchmark model (original dataset, resize, and no augmentation). Improvements were negligible for NPV, they moved around 0.02 for TPR and 0.07 for F1-score.

A generally positive augmentation effect was observed when training ResNet-18 (Table 9), both on the original and the extended dataset, with the improvements being higher than in the case of Inception-v3. With the original dataset, when using resize, applying all augmentation methods we considered proved to be the most suitable, raising the F1-score from 0.691 to 0.856, an increase of 0.165. When using edge padding, augmentations without intensity were the best, resulting in an F1-score of 0.818 (compared to a 0.684 benchmark). These combinations had the highest impacts on TPR and NPV, as well.

For training on the extended dataset, applying all augmentation methods caused the biggest improvement in combination with resize and augmentation methods without noise when using edge padding. This corresponds to the results obtained for training Inception-v3. There was a significant improvement in the F1-score from 0.675 to 0.797 and from 0.616 to 0.823 with resize and padding, respectively. NPV also increased from 0.751 to 0.833 and from 0.701 to 0.848. The biggest improvement was observed for recall, which increased from 0.561 to 0.734 and from 0.465 to 0.745, respectively. Of all experiments, we achieved at most a 0.16 increase in F1-score and TPR and 0.07 in NPV compared to the benchmark (original dataset, resize, no augmentation used).

B-line detection showed poorer results when no augmentation was used, and especially recall remained low for both models. All data augmentation combinations showed a positive effect, as can be seen in Tables 10 and 11, but for a few exceptions. The combinations that worked best were similar to those determined with A-line detection, namely all augmentation methods and augmentation without the introduction of noise.

For the Inception-v3 model with the original dataset, all augmentations improved all metrics most when combined with resize. F1-score increased from 0.531 to 0.683, NPV from 0.866 to 0.925. The biggest increase was observed for TPR, which changed from 0.196 to 0.566 thanks to introducing augmentation to the training process. Leaving out noise helped when using edge padding, with F1-score reaching 0.694, TPR at 0.588, and NPV at 0.929. A combination of only basic augmentation methods led to a decrease in all considered metrics. When training on the extended dataset, augmentation without noise worked best with resize, and augmentation without intensity led to the highest increase in performance with edge padding. F1-score was around 0.8 in both cases, which constitutes the highest value achieved for B-line detection. This represents a notable improvement compared to the benchmark model, which had an F1-score of 0.531. Best recall and NPV were achieved with the extended dataset and augmentation without noise, at 0.737 and 0.953, respectively.

Augmentation had a positive effect on training ResNet-18 for B-line detection as well, with both datasets. On the original dataset with resize, there was an increase of 0.1 in F1-score when data augmentation without noise was used (0.572 compared with 0.473), however, TPR and NPV were the highest when a combination of all augmentation methods was used. The increase was most significant with recall, with the value reaching 0.365 with augmentation compared to 0.082. With edge padding, a combination of all augmentation methods led to the best F1-score with 0.643 (up from 0.527), while other metrics reached their highest value with augmentation without noise. Recall once again more than doubled from 0.215 to 0.457.

On the extended dataset, augmentation without noise worked best in both test scenarios with different image transformations. When combined with resize, F1-score reached 0.731 (largest overall with ResNet-18 for B-line detection), TPR was at 0.686 (up from 0.329), and NPV at 0.941. With edge padding, F1-score was at 0.705, TPR at 0.73, and NPV at 0.95, the latter two constituting the best results obtained with ResNet-18 for the given task. F1-score was similarly at 0.705 when all augmentation methods were used, but the other two metrics were lower with 0.633 and 0.927, respectively.