PATMA: parser of archival tissue microarray

Background

In modern pathology, the use of tissue microarrays (TMA) is a common practice (Kallioniemi et al., 2001; Ilyas et al., 2013; Rexhepaj et al., 2013). TMA enables researchers to extract small cylinders of tissue, called cores or punches, from histological sections and arrange them in an array on a paraffin block so that multiple samples can be processed simultaneously. From this combined paraffin block slices can be cut and processed like any other histological section. TMA allows a more efficient way to analyse tissue sections, especially when they should undergo further processing, such as staining. It is possible to use immunohistochemistry (Neuman et al., 2013; Korzynska et al., 2010; Lopez et al., 2014), in situ hybridization (Garcia-Rojo, Bueno & Slodkowska, 2010), or fluorescence in situ hybridization (Du & Dua, 2010) on all cores simultaneously. This way, it lowers the intralaboratory variability of different staining concentrations between slices. Furthermore, it significantly improves processing time, allows obtaining more reproducible results, and greatly reduces the cost of analysis.

TMA are mostly used for cancer tissue evaluation. The analysis of TMA is very similar to regular analysis of histological slides (Seidal, Balaton & Battifora, 2001) as it is mostly based on visual examination, but instead of one specimen there are many in one slide. Nevertheless, reading big arrays of cores can be burdensome and requires caution. As the cores are not labelled directly on the slide with any text or number, the only way to distinguish specimens is based on their arrangement. The assumed grid structure (array) of the TMA with cores located on the grid nodes is often distorted through the mechanical procedures performed during the glass slide preparation (Fowler et al., 2011; Werner et al., 2000). It is very important to distinguish the correct order that was executed while the TMA was prepared.

Since computer-aided observation and analysis of tissue samples is increasingly common idea and TMA slides can be treated as normal histological slides, it is also possible to make their digital copy. Slide scanners create a virtual image of the whole glass slide (Kayser et al., 2006; Kayser et al., 2010; Pantanowitz et al., 2013). Whole slide images of TMA are enormous, in the context of size, pixels (resolution), as well as disk space. It is impossible to process these images at full resolution due to limited memory capabilities of a standard computer workstation. Splitting the whole virtual slide into separate images of individual cores is required for further image processing. The size of these images is suitable for analysis using a 64-bit based computer systems. Therefore, the image-processing algorithms (Redondo et al., 2012; Roszkowiak et al., 2016; Markiewicz et al., 2006; Neuman et al., 2010; Korzynska et al., 2013) process cores rather than whole slides.

Although TMA is gaining popularity in clinical practice, it is still more common in research facilities. A vital step to improve the speed and quality of this process is to correctly locate each tissue core in the array. Usually the tissue cores are aligned in a typical microarray, but quite often the TMA cores are askew and incomplete. The problems with automatic selection of cores in TMA include: places where the core should be located, but is actually absent because of technical issues; incomplete cores; cores consisting of many fragments; heavily skewed rows, where cores are relocated in unpredictable manner; stain deposits in the background.

Aim

The main aim of this study was to develop a computerized procedure capable of automatically finding and extracting cores from images of TMA. This software was developed on special request from Molecular Biology & Research Laboratory from Hospital Verge de la Cinta (Institut d’Investigacio Sanitaria Pere Virgili—IISPV), located in Tortosa, Spain. It was named PATMA for Parser of Archival TMA. The procedure is capable of handling core sections under standard conditions—moderately aligned array with well contrasted cores from the background. Once the TMA cores are located, they are stored in separate files for further assessment.

Related works

Currently, there is lack of available software that helps the researcher to extract single core image from the whole slide image, which is free of charge, simple to use, and automatic.

There are a few scientific works (Lahrmann et al., 2010; Dell’Anna et al., 2005) that can be related to this kind of image processing of TMA. Most of the software focuses on the designing (Thallinger et al., 2007) or scoring punches of TMA (Amaral et al., 2013). Some have been created to support specific study and do not support various data. Other programs apply manual processing (Chen, Reiss & Foran, 2004). The user must set the number of rows and columns and sometimes select the first core manually. Web-based solutions (Della Mea et al., 2006; Demichelis et al., 2006; Galizia et al., 2008) are not very efficient while working with high-resolution whole slide image, because transferring files with a size of 30 gigabytes or more is rather problematic. Some programs are meant to work with computer-controlled microscope for collecting data (Rabinovich et al., 2006); with addition of TMA planning program this may give good results for new research but not while handling archival data. Other solutions found in the literature include Image Miner (Foran et al., 2011), recently published complex computer-aided diagnosis system (Milagro Fernandez-Carrobles et al., 2014; Milagro Fernandez-Carrobles et al., 2015) for TMA, and others (Morgan et al., 2004; Liu et al., 2005).

There are also commercial solutions such as TMALab from APERIO or TMADesigner2 from APHELYS. The major disadvantage of commercial software is that it is bound to a specific slide scanner.

File reading

Virtual slide files usually have an inner structure of tiled images that can form the full picture in multiple resolutions. Loaded files are investigated for available resolutions concealed inside. Depending on the user’s choice—manual or automatic—working resolution is selected for analysis. Automatic selection always chooses the smallest resolution. Manual selection gives user a choice by providing all available resolutions. The selected image is then preprocessed. As of now, the supported format for virtual slides is single-file pyramidal tiled TIFF file format.

Preprocessing

The aim of this preprocessing is to create a binary mask with punch location. First, the image is converted from RGB to Lab colour space. Image is binarised with threshold value defined as most common intensity in Luminescence layer (Lab colour space) decreased by 5. Then, the morphological area opening is executed. Depending on the selected working resolution, objects containing fewer than 1,000 pixels are removed from images smaller than 10⁷ square pixels and objects fewer than 10,000 pixels from larger images. Next, a series of other morphological operations including closing, filling, and border clearing are performed (see Algorithm 1). As a result of the preprocessing we obtain black and white images where objects correspond to punch (or fragment of punch) location.

Occasionally, strong stain deposit in the background or overstained tissue sections cause extremely unsatisfactory results. In such case, user can choose expert mode and manually modify threshold value as well as size limit of the object to be treated as punch and size of structural element used for morphological operations. If the TMA image was loaded with a wrong orientation, the possibility of rotating the whole TMA is also available in expert mode section. Through this expert preprocessing, a binary map can be previewed by the user to adjust the parameter values to a particular image.

Punch handling

We collect properties of found objects (area, centroid, eccentricity, and bounding box—standard MATLAB features). Then, we try to assess the common distance between the cores. This reference distance (refDist) is calculated based on the most circular punch (object with the minimum value of eccentricity) as halved mean of bounding box’s width and height. We assume that the distance between the punches is no greater than reference distance. We also assume that punches are objects with eccentricity lower than 0.9. Objects that fulfill the criteria are assigned to columns and rows based on their centroid coordinates (see Algorithm 2).

First, objects are assigned to columns one after the other. Among all the unassigned objects we seek the one with the smallest horizontal coordinate. Then, we look for objects with centroid located not further than reference distance, but only in terms of horizontal coordinate. The valid objects are assigned to currently evaluated column. Then, we look for objects in the next column.

After objects are allocated to columns, we try to determine the proper row assignment. First step is similar to that in column search; the object with the lowest vertical coordinate is found. Then, we examine the rest of the objects if they are valid for this row. Objects within the reference distance in terms of vertical coordinate are assigned to currently evaluated row. If this criterion is not met, we have an additional condition based on the centroids of objects already assigned to the currently examined row. We evaluate the polynomial of second degree based on at least three points. Then, we make an extrapolation of that polynomial with the horizontal value of the currently examined centroid. If the assumed point is within the reference distance of examined centroid, then it is also assigned to the current row. Example of such assignment is shown in Fig. 2.

Then, objects with a missing value of row (or column) are managed. The distance between evaluated object’s centroid and the nearest object’s centroid is calculated. If it is greater than the reference distance, analyzed object is assigned to the adjacent row (or column). However, for objects closer than the reference distance, closest objects are combined/merged into one. Whenever object has no row and column value, a mean of coordinates of each column and row is calculated. Then, this object is assigned to the closest row and column by distance.

Finally, duplicate assignments to rows and columns are treated with the same algorithm. Around every object a region of interest (ROI) similar to the bounding box is created. The regions are adjusted so that there is no overlap. Lastly, correct numerical labels, corresponding to TMA punch numbering, are applied to each ROI.

The designed array structure of the TMA is often distorted, cores can be relocated and may also disappear completely leaving “holes.” Because we evaluate assumed reference distance between the cores of TMA, we can figure out where the cores should be in the array despite their actual absence. We assign successive numerical label to the identified “holes” (ROI object with turned off Validity option—see the following paragraph).

User input

The ROI selected in the previous step are presented as an overlay in the processed image, as in Fig. 3. At this stage, the user may apply any necessary corrections. Moving and resizing of selected ROI is possible, because it is draggable. Manual corrections of numerical labels can be applied. Moreover, there is an option of addition and deletion of ROI.

To make editing easier, there is a table presented in the GUI that consists of Numerical label, Position parameters, and Validity. Validity option provides a way to keep correct numbering of punches while there are empty spaces between them. Objects marked as not valid are not extracted in the final step. All of these attributes can be manually changed.

Extraction

The selected ROI containing punches are extracted from the maximum available resolution image as separate files. The user can add border margin to the extracted punch to be sure that every pixel with information is inside the selected ROI. Only one punch or all punches from the TMA can be extracted. Images are saved with easily distinguishable filenames in TIFF format.

Future work

In the future, we want to further develop the software to improve its performance. We plan to implement the background elimination algorithm in the preprocessing stage to cope with strong stain deposits. We would like to improve punches’ assignment to correct rows and columns, even in heavily skewed tissue microarrays, for example by assuming grid structure that would be matched elastically to actual structure. Finally, we plan to expand the supported file types, so that the other file formats would be handled.