How reproducible are the measurements of leaf fluctuating asymmetry?

Samples and participants

For this study, we used leaves of Betula pubescens Ehrh. (Betulaceae), because this tree is commonly used for studying environmental impacts on leaf FA (more than 50 papers are published to date). About 200 leaves with no traces of mechanical damage, deformation or insect feeding were collected on July 21, 2014 from a single tree of B. pubescens near Turku, Finland. The leaves were pressed between the sheets of filter paper and dried as ordinary herbarium specimens.

A selection of 100 perfectly preserved leaves was haphazardly divided into 10 samples, each sample (consisting of 10 leaves) was scanned with high resolution and the scale was added to each image. All 10 images were sent to 31 scientists who agreed to participate in the project (for the list of participants, consult Appendix S1). All these scientists have published studies addressing FA in plants and were therefore experienced in FA measurements.

The participants were numbered in the order they took part in the study, and these numbers were used to identify participants across the study. The participants were asked (a) to measure the width of the left and right halves of each leaf at the midpoint between the base and the apex of leaf lamina (WL and WR; Fig. 1) using the same instruments and methods as they had used in their earlier studies and (b) to provide detailed information about the instruments and methods used in the measurements of the test samples, as well as in their earlier studies of plant FA. The character selected for this study is commonly used to quantify FA of plant leaves (more than 200 papers are published to date).

Data analysis

Preliminary analysis demonstrated that the results by one participant greatly differed from the results of all other participants. The participant was informed about the absence of correlations between his results and the results of other participants and was asked to re-measure the test sample. The results of the first measurement were excluded from the subsequent analyses.

The values of WL and WR reported by the participants were checked for the consistency of scale. The units of measurements were converted from cm to mm whenever necessary, and all values in three of 31 data sets were multiplied by the correction coefficient to remove the systematic differences in leaf width among the participants. The mixed model ANOVA (with leaf side considered as a fixed factor and individual leaf as a random factor; SAS MIXED procedure: SAS Institute, 2009) was used to test the entire pool of data for the significance of FA relative to measurement error and for the presence of directional asymmetry (as described by Palmer & Strobeck, 1986; Palmer & Strobeck, 2003; Merilä & Björklund, 1995). The reproducibility was quantified (a) by calculating the index ME5 = [MS_i − MS_m]/[MS_i + (n − 1)MS_m], where MS_i and MS_m are the interaction and error MS from a sides × individuals ANOVA (Palmer & Strobeck, 2003), and (b) by using gauge repeatability and reproducibility (GRR) analysis (SAS VARCOMP procedure), which allows partitioning of total variance into variance of the monitored process and variance of the measurement system (Burdick, Borror & Montgomery, 2003).

The principal component analysis (PCA; SAS PRINCOMP procedure) of the signed values of FA, calculated as follows: FA = 2 × (WL − WR)/(WL + WR), was used to explore relative similarities and differences between the results obtained by the participants. The reasons for the differences among the groups of participants were identified by correlating the extracted vectors of the two first axes identified by PCA with the precision of measurements, the time spent on measurements and the numbers of leaves that were measured earlier by the researchers (SAS CORR procedure). The distribution of the absolute values of FA was greatly skewed due to a large (19.3%) numbers of zeros; therefore, the median values were compared among participants using the Kruskal-Wallis test (SAS NPAR1WAY procedure; SAS Institute, 2009).

Instruments and protocols used to measure test samples

The data acquisition protocols applied by the participants differed in many minor but important details (Appendix S2). More than half of the participants used rulers, dividers and digital callipers to measure the width of leaf halves from the printed images of the leaves. Other participants measured leaf images using different software programs (listed in Appendix S2), including the original software that had been specifically created for the measurements of leaf FA. Only roughly half of the participants used instrumental methods to assure perpendicularity of their measurement lines with the midrib. Five participants identified the place of measurements visually, rather than by measuring leaf length and finding the exact location of the middle of the leaf lamina. Roughly half of the participants selected the starting points of their measurements away from the median line of the midrib, on the external margin of the midrib—thereby excluding the midrib width from the leaf width measurement. Most participants measured the width of the left and right halves of a leaf sequentially, but some participants obtained these measurements simultaneously, by placing the zero point of the ruler on the left leaf margin and reading two measurements—the distance to the midrib and the distance to the right leaf margin (Appendix S2). The measurement protocols applied by the participants clearly differed in many other peculiarities, which were impossible to uncover because the participants generally described their instruments and methods in insufficient detail. Some important differences in data acquisition protocols were revealed just by chance.

Seven participants slightly modified the protocol of leaf measurements. Participant 14 measured the distance from the tips of the teeth that were closest to the middle of the leaf to the point where the straight line connecting these teeth crossed the midrib. Participant 2 measured the width of the left side of a leaf in the middle of leaf lamina perpendicular to midrib, but adjusted the end point for the measurement of the right side of a leaf in such a way that this point was selected in the same position relative to marginal teeth as the end point on the left leaf margin. Participant 1 did not purposely adjust the positions of the end points relative to the teeth on leaf margins, but appreciated that this adjustment may have been done unintentionally. Five participants measured a leaf not at the midpoint between its apex and base, but at the place where the leaf width attained its highest value (Appendix S2).

Only three participants reported that the test samples were measured twice by different observers. Another 13 participants reported two measurements by the same observer, and the remaining participants conducted the measurements only once.

Methods used for data collection in published studies

Two-thirds of the participants measured FA from fresh plant leaves or from scanned images of fresh leaves; one participant measured the images of leaves that were fixed in ethanol; and the remaining participants measured pressed and dried leaves or their images. The participants who worked with fresh leaves often bent them across the midrib (in such a way that the apex of a leaf coincided with the base) to mark the line along which the leaf halves were then measured. The participants, with nearly equal frequencies, reported long-term storage of the measured leaves, of their images and no archiving at all (or storing the leaves only until the results were published; Appendix S2).

Only three of 31 participants consistently used a blind method in their earlier studies in such a way that the persons who conducted measurements were not aware of the origin of the measured samples and/or of the hypothesis being tested.

Identification of FA

Analysis of all measurements reported by participants (Appendix S3) demonstrated that the variation in width among the 100 measured birch leaves was highly significant (F_99,99 = 45.9, P < 0.0001). The left and right sides of birch leaves did not differ in width (F_1,99 = 0.60, P = 0.44), indicating an absence of directional asymmetry, whereas the highly significant side × leaf interaction (F_99,6000 = 3.46, P < 0.0001) confirmed the existence of FA.

Variation among the reported values of FA

The values of FA obtained from the measurements of the same set of 100 birch leaves greatly varied among the participants (χ² = 169.5, d.f. = 30, P < 0.0001): the median values ranged from 0.000 to 0.074 (Fig. 2), decreasing with the increase in the precision of measurements (r_S = − 0.50, n = 31, P = 0.004).

The measurements conducted on leaf images using different software yielded the same median values of FA as obtained for the measurements conducted from the printed copies of leaf images using rulers or callipers (χ² = 1.45, d.f. = 1, P = 0.22). Other recorded characteristics of the measurement protocol (Appendix S2) also had no statistically significant effect on median values of FA (P > 0.10).

The median values of FA did not depend on the time spent obtaining the leaf measurements (r_S = 0.12, n = 30, P = 0.54) or on the experience of the participant (quantified by the number of leaves measured in the course of the earlier studies; r_S = 0.05, n = 30, P = 0.77).

Similarities and dissimilarities among the results of measurements

The first two axes of the PCA accounted for 52.3% of the total variability among the FA values calculated from the measurements made by 31 participants (44.8% of variability along the first axis and 7.5% along the second axis; Fig. 3). The median value of the Pearson correlation coefficient between FA values based on measurements by different participants was surprisingly small (0.41), and 4.8% of all pairwise correlations were not statistically significant.

The extracted vectors of the PC1 axis were best explained by the precision of measurements (r = 0.44, n = 31, P = 0.02), and those of the PC2 axis by the time spent in measuring a test sample (r = 0.40, n = 30, P = 0.03). The visual inspection of Fig. 3 suggests that the participants can be divided into two groups that differ in their values of PC2.

Reproducibility of the results

The participants reported highly variable signed values of FA of individual leaves (Fig. 4). In particular, 97 of 100 leaves were reported as perfectly symmetrical (FA = 0.000) by at least one participant; the leaf-specific coefficients of variation of FA ranged from 25% to 179% (median value 84%). Nevertheless, the variation in FA among individual leaves, calculated from the results of measurements by 31 participants, was highly significant (χ² = 595.4, d.f. = 99, P < 0.0001). However, the reported values of FA decreased with leaf size (r_S = − 0.26, n = 100, P = 0.009); therefore, at least a part of the detected variation in FA is explained by variation in leaf size.

The overall reproducibility of the results among the participants (evaluated by the index ME5, according to Palmer & Strobeck, 2003) appeared to be very low (0.074). The GRR analysis demonstrated that the variance of the measurement system (i.e., among the participants) accounted for 79.5% of the total variance, and that the discrimination ratio—another index reflecting the quality of the measurement system—was 1.39 (CI₉₅ = 1.28…1.57). Exclusion of the data provided by seven participants who modified the measurement protocol (see above) increased the reproducibility of the results (ME5 = 0.096; variance of the measurement system 59.6%; discrimination ratio 2.07).