Measurement error using a SeeMaLab structured light 3D scanner against a Microscribe 3D digitizer

Introduction

Geometric morphometrics (Bookstein, 1998; Rohlf & Marcus, 1993; O’Higgins, 2000; Adams, Rohlf & Slice, 2004; Slice, 2005; Mitteroecker & Gunz, 2009) has proven a valuable tool in ecological and evolutionary studies of e.g., phenotypic development (Polly, 2008) and population structure (e.g., Galatius, Kinze & Teilmann, 2012). Within the field of geometric morphometrics, there has been great interest in the development and availability of equipment for collecting 3D data. It is an acknowledged and widely used practice to obtain landmarks directly from a physical specimen through a three-dimensional (3D) digitizer arm—most commonly a Microscribe digitizer. In recent years, however, collecting landmarks on a 3D digital model, obtained by 3D scanning the physical specimen, has become a viable and widely accepted alternative. Such a pipeline offers many important advantages, namely the creation of a permanent digital model of a specimen, which can be stored, easily shared with other researchers, and used for further analyses or adjustments of the measurement protocol. Furthermore, the full digital 3D model offers the possibility to analyse shape using more advanced statistical methods compared to the analysis of the collected landmarks using a digitizer.

3D surface scanners create high-density point clouds of the scanned object’s surface. They are portable, highly accurate and non-destructive, and are thus an obvious choice for 3D digitizing physical specimens, such as skulls and bones in museum collections. To confidently replace digitizer arms with 3D surface scanners for geometric morphometrics, it is important to assess the measurement error introduced by using 3D surface scanners and to compare the two devices.

Landmarks can be classified into three distinct types based on their anatomical locations and biological significance (Bookstein, 1991). Previous studies have compared the precision of Type I, II and III landmark measurements obtained with a digitizer and from surface scans of human crania. One study concluded that landmark measurements obtained with the two devices are comparable for clearly observed landmarks (Algee-Hewitt & Wheat, 2016), whereas another study revealed that landmark coordinates obtained with the digitizer are more precise on average, although the precision of the two devices varies considerably depending on the landmark type (Sholts et al., 2011). However, none of these studies investigated or quantified different sources of measurement error.

In 3D geometric morphometric studies, measurement error can be introduced at any phase, by several sources (Fruciano, 2016). There is measurement error from replicability of landmark placing (within-operator), and due to the involvement of multiple operators (between-operator). Technological sources of measurement error include the use of different devices (between-device), but also cover 3D model creation steps, such as scanner resolution and algorithms used for fitting and smoothing. Presentation error involves differences in the way specimens are positioned during data acquisition, and it includes measurement error due to optical distortion.

The only study which has addressed measurement error in geometric morphometric data collection involving both a digitizer and a surface scanner that we are aware of is Robinson & Terhune (2017). These authors suggested that measuring landmarks based on 3D scans slightly increases repeatability and landmark precision compared to measurements with a digitizer, and that between-operator error is greater than between-device error, while within-operator error is similar among devices. However, landmark coordinate measurement is subject to a compounding error when using a digitizer, as an operator and the digitizer interact during the data collection. When comparing digitizers to surface scanners, it is therefore important to introduce the same type of error, which can be achieved by letting each operator separately scan the specimens, then letting them place landmarks on the reconstructed 3D models of their own scans.

In addition to surface scanners, there are other important 3D capture modalities that differ in their properties and come with specific advantages and disadvantages. Highly accurate micro-computed tomography (µCT) scanners generate a 3D volumetric model, and have been seen as the “gold standard” of 3D shape capture technology (e.g., Marcy et al., 2018). µCT scanners capture more information, such as internal structure, however, they are expensive and not portable, and the scanning process is usually time-intensive. Typically, µCT scanning will give a volumetric image where the number of voxels is proportional to the size of the detector. Thus, in order to image a large object in high resolution, multiple scans must be acquired and stitched together. The resolution is limited by the spot size of the X-ray (or neutron) source and by the physical setup of the imaging system. Moreover, µCT scanning might potentially lead to DNA damage (Bertrand et al., 2015; Grieshaber et al., 2008). Marcy et al. (2018) found that using surface scans generates more random within-operator error compared to µCT scans, however not accounting for multiple operator error, whereas Shearer et al. (2017) and Giacomini et al. (2019) found consistent operator error rates for surface and µCT scans.

Another inexpensive, portable and increasingly popular 3D capture modality is photogrammetry, which is based on aligning multiple 2D photographs into a 3D model. Previous studies found that photogrammetry can be seen as a reliable alternative to surface scanners, both in case of large skulls (Katz & Friess, 2014; Evin et al., 2016), and for smaller skulls, where the use of photogrammetry has only been assessed at interspecific level (Giacomini et al., 2019). However, photogrammetry involves time-consuming image processing compared to laser surface scanners (Katz & Friess, 2014), and it involves a higher risk of inconsistency or human error during photography (Marcy et al., 2018). Giacomini et al. (2019) report a lack of detail achieved for teeth reconstruction based on photogrammetry as well as difficulties in reproducing thin structures, however, these issues can also occur when using surface scanners.

The presence of high random measurement error can obscure the biological signal under investigation through an increase in the amount of variance (Arnqvist & Mårtensson, 1998; Fruciano, 2016). As a consequence, measurement error is more serious when the biological variation of interest is relatively low. At interspecific level, most studies combining landmark data collected using different devices found that measurement error was sufficiently small compared to the biological variation (Robinson & Terhune, 2017; Marcy et al., 2018; Giacomini et al., 2019), whereas one study identifies a between-operator error similar to the level of biological variation (Shearer et al., 2017). At intraspecific level, Robinson & Terhune (2017) found that the biological variation is obscured by all considered types of measurement error. However, the exclusion of difficult to place landmarks can substantially reduce the amount of measurement error to low levels compared to intraspecific variation (Fruciano et al., 2017).

In addition to random measurement error, systematic error can be present in geometric morphometric datasets, changing the mean and introducing a bias to the data. As a result, differences induced by systematic error might be interpreted as differences between groups in the analysis (Fruciano, 2016). A few studies confirmed the presence of small amounts of bias considering different 3D capture modalities (Fruciano et al., 2017; Marcy et al., 2018), and 2D landmark data (Fruciano et al., 2019). However, none of the studies comparing a digitizer with 3D capture modalities for morphometric data collection have examined the systematic error introduced by device and operator.

The purpose of our study is to examine whether 3D surface scanners confidently can replace digitizers for geometric morphometrics. To this end, we quantified the levels of measurement error introduced by analysing 22 grey seal skulls using a Microscribe digitizer, and a 3D structured light scanning setup (SeeMaLab (Eiriksson et al., 2016)) for landmark measurement. Specifically, we assessed whether specimen shape can be sufficiently captured by the SeeMaLab scanner compared to a Microscribe digitizer, and we compared the level of measurement error to the biological variation at intraspecific level. In contrast to previous studies, we accounted for the compounding error of observer and machine, which arises when using a digitizer, by making different scans and thus 3D models of every specimen for the two operators. We further investigated whether there are systematic differences between the Microscribe digitizer and the SeeMaLab scanner, and between operators. The SeeMaLab scanner is assembled from low-cost, off-the-shelf parts, and it is an open source system (Technical University of Denmark, Kongens Lyngby, Denmark; https://eco3d.compute.dtu.dk/pages/3d_scanner; Eiriksson et al., 2016). In order to assess the level of error introduced by our scanning setup relative to other sources of measurement error, we additionally analysed an extended scanner dataset, where both operators were collecting landmark measurements on both reconstructed 3D models.

Materials and Methods

Landmark data collection

For each of the 22 skulls, Cartesian coordinates of 31 fixed anatomical landmarks of Type I and II (Fig. 1; Table A2) were recorded by two operators. The choice of landmarks is inspired by previous morphometric studies of ringed seal (Pusa hispida) and Antarctic fur seal (Arctocephalus gazella) skulls (Amano, Hayano & Miyazaki, 2002; Daneri et al., 2005). Each of the two operators applied two different measurement methods: Direct measurement on the physical skulls using a Microscribe 3D digitizer (Immersion Corporation, San Jose, CA, USA; https://www.immersion.com; Fig. 1 in Article S1); and placement of the landmarks on the reconstructed 3D digital models of their own scan using the Stratovan Checkpoint software version 2018.08.07 (Stratovan Corporation, Davis, CA, USA; https://www.stratovan.com/products/checkpoint). As none of the landmarks were considered difficult to see or reach during data collection, landmarks were placed from a single perspective with the Microscribe digitizer. When using the Microscribe digitizer, the errors caused by the device itself and the operator are combined. Thus, in order to obtain comparable datasets from the Microscribe digitizer and the scanner pipeline, we made each operator digitize their own scans, combining any associated errors for the same observer in the scanner pipeline. Moreover, in order to assess within-operator error, each measurement session was completed twice by both operators. We considered two repeated measurements to be sufficient for estimation of raw, direct measurement errors. When using the Microscribe digitizer, landmark measurement on the same skull was repeated after a break of about twenty minutes, whereas in case of the scanner pipeline, landmark annotation was only repeated after having annotated at least two other specimens. The datasets obtained by operator A using a Microscribe digitizer were collected about one year earlier than all remaining datasets. This resulted in eight datasets, each containing landmark configurations from all 22 skulls, giving a total sample of 176 configurations of landmarks. We call the entirety of these datasets Device Comparison Dataset. Figure 2 gives an overview over the data acquisition process.

Finally, each operator was also annotating the reconstructed 3D models based on the scan from the other operator twice. Thus, for all specimens, we collected landmark measurements on the 3D models based on an operator’s own scans, and on the 3D models based on the scans from the other operator. We call the entirety of these scanner datasets Extended Scanner Dataset (Fig. 3). In contrast to the data collection using the Microscribe digitizer, which is done directly on a physical skull, there are two steps involved when using the scanner pipeline: reconstruction of a 3D model, and placing landmarks on this digital model. Using the Extended Scanner Dataset allows us to disentangle the scanning-related error from the observer measurement error, and assess the level of error introduced by our scanning setup. A detailed data collection protocol is included in Article S1, allowing replication of our measurements.

Results

We found that the distribution of Procrustes distances between devices is comparable to that between operators (Fig. 4). In general, Procrustes distances between devices or between operators exhibit larger values than those within operators or between scans. Moreover, Procrustes distances between scans are distributed similarly to those within operators. The median of the Procrustes distances within operators is similar for both devices, providing evidence that the digitizer and scanner exhibit similar precision. However, there is a longer tail for the scanner-based data. The Procrustes distances between operators using the scanner pipeline seem to be smaller than when using the digitizer. Measurement differences between devices seem to be similarly distributed for both operators. Our analysis further shows that the median of the Procrustes distances between specimens of grey seal is three to six times larger than the medians of Procrustes distances corresponding to different measurement errors. There is a slight overlap between the smallest 25% of Procrustes distances between specimens and Procrustes distances corresponding to different measurement errors, which is almost completely associated with outliers of the latter.

There are several outliers corresponding to substantial measurement error in all comparisons (Fig. 4). An analysis of these outliers revealed that they are mainly caused by measurements of a few landmarks on six skulls, where three specimens alone (96, 42.23, 14) are responsible for 87.5% of the outliers. On these specimens, some of these landmarks were difficult to place both on the 3D models and on the physical skulls (e.g., landmarks 1, 12/26, 13/27, 14/28, 17/31), as was noted during data acquisition. Our analysis further revealed that the observed landmark measurement errors of the outliers are due to both differences between operators and devices, or a combination of them. We provide some outlier examples in Figs. S1 and S2.

We obtained corresponding results in our analysis based on pairwise Procrustes ANOVAs where we assessed the level of measurement error (Table 1). The most substantial error sources are between-device and between-operator, which are comparable in size to each other. On average, measurement differences between devices explain 4.6%, and between operators 4.5%, respectively, of the total shape variation among datasets in pairwise comparisons. Moreover, error between scans is similar to within-operator error, as they explain 2.0%, and 1.4%, respectively, of pairwise total shape variation on average. The repeatabilities computed for the comparisons by error source exhibit a similar pattern.

Running a Nested Procrustes ANOVA on the whole Device Comparison Dataset, thereby testing the two devices on equal terms, again, gave a comparable result. We found that the device accounts for 2.5% of total shape variation, while between-operator and within-operator error explain 3.7% and 1.4% of the total shape variation (Table 2A). Most of the total shape variation (92.4%) is explained by biological variation among grey seal specimens.

Computing Nested Procrustes ANOVAs for both devices separately, we found that the variation in residuals contributes to a similarly small fraction of total shape variation for both devices (Tables 2B and 2C). As the residuals correspond to differences in landmark replica (i.e., within operators), this indicates that the precision is similar for both the digitizer and the scanner pipeline, and high for grey seal skulls. This result is also reflected in the repeatabilities for landmark replica only, which is 0.983 for the scanner-based datasets, and 0.989 for the digitizer-based datasets, respectively. We again find evidence that the variation between operators contributes slightly more to total shape variation among digitizer-based datasets (4.7%) than among scanner-based datasets (2.8%).

Running a Nested Procrustes ANOVA on the Extended Scanner Dataset, thereby disentangling the scanning-related error from the operator error, we found that out of all identified error sources, scanning-related error is smallest and explains only 0.7% of among-dataset shape variation (Table 3).

Our analyses on general shape bias, which are based on the residuals computed by subtracting specimen means from the individual landmark configurations in the Device Comparison Dataset, visually indicate that both device and operator introduce bias, i.e., non-random error (Fig. 5). The scatterplot of the scores along the first two principal components (Fig. 5A) on the residuals from specimen means shows that the scanner pipeline and digitizer methods can be roughly linearly separated along the first principal components. Moreover, in case of the digitizer method, one can roughly categorize operator A and B along the second principal component, whereas the scores of operator A and B are slightly more overlapping for the scanning pipeline. This is further evidence that the operator-bias is larger on the digitizer than on the scanner. Between devices, we find the largest differences in average shape (Fig. 5C) for landmarks that are less clearly defined, including e.g., landmarks at the alveolar margins of the teeth (e.g., 22), caudal landmarks (e.g., 17/31, 18), and around the orbits (e.g., 12/26, 14/28). We further observe that for symmetric pairs of landmarks, the between-device error is not reflecting this symmetry: In two-thirds of the cases, the error is larger on the lateral left side. Moreover, we find evidence that on the 3D model, the surface spanned by landmarks 2–6, which are located on the sutures of the nasal, is translated inwards from the analogous surface measured using the digitizer. Between operators (Figs. 5D and 5E), larger shape differences are predominant in similar regions as between devices, however, with less bias at the alveolar (teeth) landmarks. In general, between-operator shape differences again seem to be larger using the digitizer. For the scanner pipeline, the smallest between-operator bias is for landmarks located on the sutures of the nasal (all Type I), which is in contrast to landmark measurement using a digitizer. Similar to between-device error, we note that between-operator error does not seem to be symmetric for some symmetric landmarks (12/26, 13/27 and 17/31) when using the digitizer.

Discussion

In this study, we investigated whether a 3D structured light scanning setup is a viable alternative to a Microscribe digitizer, one of the most popular choices for fixed landmark measurement of skeletal material within geometric morphometrics, by assessing measurement error involved in data collection on grey seal skulls. We specifically accounted for the interaction between operator and digitizer during data collection by introducing a comparable compounding effect between operator and machine for the scanner pipeline. Without doing this, we might have introduced bias in favour of the SeeMaLab scanner over a digitizer. In our analysis, we compared within- and between-operator error obtained with the two devices. Additionally, we quantified the variation introduced by measurement device and compared it to other sources of variation. Furthermore, we illustrated that both measurement method and operator introduce general shape bias (i.e., non-random variation in mean shape). Finally, we quantified the scanning-related error (i.e., the error introduced by small differences in the scanning pipeline).

Overall, our analysis suggests that the SeeMaLab surface scanner provides a good alternative to the Microscribe digitizer for geometric morphometrics. We found comparably small within-operator errors, which implies that the precision of the SeeMaLab scanner compares with the precision of the digitizer. This was also reflected in the repeatability scores for landmarking replica (scanner: 0.98, digitizer: 0.99), which were similar to previously reported values obtained on human skulls (digitizer: 0.99; Badawi-Fayad & Cabanis, 2007), kangaroo-size skulls (overall repeatability for laser scanners and photogrammetry (pooled): 0.78−0.96; Fruciano et al., 2017), and bat skulls (laser scanner, µCT scanner and photogrammetry (pooled): 0.97; Giacomini et al., 2019). Marcy et al. (2018) have found a smaller repeatability of 0.64 for laser scanner measurements of fixed landmarks using small rodent skulls.

From the distribution of Procrustes distances, which were computed between landmark replica measured by the same operator using the same device, we could see that for many skulls, operators were more consistent in placing landmarks on the 3D digital model than when digitizing the physical skull. However, the long tail of the distribution for the scanner-based data showed that for a few skulls, the operators were less consistent in repeating annotation of some landmarks than when using the Microscribe digitizer. The main drivers of the observed inconsistencies were a few landmarks (symmetric pairs 13/27 and 14/28), which both operators found difficult to digitize. Leaving these landmarks out, the long tail disappeared, and there was evidence that landmark placement using the scanner pipeline was more precise than using the digitizer, which is in line with Robinson & Terhune (2017). As opposed to this, Sholts et al. (2011) have found that Type I and II landmarks can be more precisely identified using a digitizer compared to a surface scanner, which might be partly explained by the fact that for some specimens, the authors reported difficulties in locating the sutures on the 3D model due to surface alterations. Our results underline the importance of carefully choosing the landmarks when using a scanner, as previously has been pointed out by Fruciano et al. (2017). Another approach could be to repeat measurements of difficult landmarks on the 3D models, as taking averages of repeated measurements reduces within-operator error (Arnqvist & Mårtensson, 1998; Fruciano, 2016).

In general, we found that between-operator error is larger than within-operator error, as previously shown by other studies (Robinson & Terhune, 2017; Shearer et al., 2017; Wilson, Cardoso & Humphrey, 2011), and that between-operator error is systematic, which is in line with Fruciano et al. (2017). Our results show that between-operator error is smaller in the scanner-based dataset than in the digitizer-based dataset, which is in contrast to Robinson & Terhune (2017), who found similar error for two such devices. Part of the reason for the difference could be that in this study, we investigated larger skulls, which resulted in relatively higher-resolution scans compared to Robinson & Terhune (2017), who reported a substantial variation in the resolution of their scans. When comparing different scanners, we would expect between-operator error to be more similar, as placing landmarks on 3D models resulting from different scanners is more similar than placing landmarks using a digitizer. Shearer et al. (2017) found that between-operator error is independent of scan type, whereas Fruciano et al. (2017), similarly to our study, reported differences. One potential explanation for the observed difference in between-operator error between devices could be that the digitizer might have been used differently by the operators in terms of placing landmarks from slightly different angles. This could have resulted in slightly different offsets as the Microscribe tip is a bit blunt. Furthermore, there might also be differences in the digitizer’s joints, which might have been used in slightly different ways by the two operators. In addition, the operators might have investigated the landmarks from a more similar angle using the scanner pipeline, as they could rotate the 3D model and zoom in, whereas an operator had to change the position of her/his body (e.g., bending down) to be able to place a landmark with the digitizer, which could be done in many different ways. In line with this reasoning, we found some evidence that the placement of the digitizer with respect to the skull had an influence on between-operator error, as we could see that between-operator error seemed to be more bilaterally different for some symmetric landmarks (12/26, 13/27 and 17/31) when using a digitizer compared to the scanner (Article S1, Fig. 1 shows our measurement setup using a digitizer). Another interesting result is that the smallest between-operator bias was obtained for landmarks located on intersecting sutures of the nasal in case of the scanner pipeline, which was not the case for landmark measurement using a digitizer. One additional explanation is that landmark coordinates were bound to the surface of the 3D model, whereas landmark coordinates measured using a digitizer might in practice have been digitized off the skull, providing a further dimension of error for the Microscribe.

We also found systematic error between the two devices. This is in line with Fruciano et al. (2017), who documented the presence of non-random error between different 3D capture modalities. Specifically, we found the largest differences in average shape for landmarks that are less clearly defined (Type II). The placement of these landmarks was to a larger extent driven by an operator’s interpretation, such that device-specific characteristics might have led to a larger bias. For example, landmarks 1 and 14/28 are points of maximum curvature located at tips of processes. An operator could locate these landmarks both by visual and tactile means on the physical skull, which was quite helpful, whereas the tactile component is lost when using a 3D digital model. Moreover, it might have been more difficult to place landmarks on a computer screen compared to using a digitizer, as the computer image is two-dimensional. Future improvements, such as the use of Virtual Reality, might make the 3D view of the digital model tangible to a level comparable to reality, while also providing all the advantages of a digital model. On a few skulls, landmarks 14/28 were located on a very pointy tip. There is no guarantee that such a thin structure is scanned properly. Interestingly, for two-thirds of the symmetric landmark pairs, the between-device error seemed to be larger for landmarks located on the left side. Similarly to the analysis of between-operator error, this asymmetry might have been caused by the placement of the digitizer relative to the skull, which was on the dorsal side. One example is landmark 22, which is located at the posterior alveolar margin of canine. However, we cannot exclude the possibility that the 3D models were biased around the teeth due to the observed problems with reflection and being out of focus during scanning. Another source of error may be exemplified by landmarks 17/31 and 18, which are caudal landmarks. In our digitizer setup, these landmarks were placed in an awkward body position where the operator was bending down, and pressing the foot pedal at the same time. This is in sharp contrast to placing these landmarks on the 3D digital model, where the operator could focus on the task in a relaxed body position as manipulation of the 3D model happened on the computer screen. We further observed that the surface spanned by the Type I landmarks 2-6 on the 3D model was translated inwards compared to the surface spanned by these landmarks when measured using the digitizer. This might be explained by the fact that these landmarks are located at intersections of bones at sutures. On the 3D model, these landmarks were automatically placed on the surface, whereas the digitizer tip did not necessarily have to touch the surface of the skull. Another explanation could be that the 3D model differed in size from the physical skull. However, when comparing various inter-landmark distances from Microscribe and scanner, we did not find any evidence for a difference in skull size. In general, the colour model of the 3D digital models is imperfect, which might also have led to differences in landmark placement between the two devices. However, the colour model could easily be improved.

Our Procrustes ANOVA results show that the factors measurement device, operator and landmark replica all are statistically significant and of similar importance, but are still rather minor sources of total shape variation, compared to the biological variation of the grey seal skulls. Our analysis of pairwise Procrustes distances revealed a similar pattern. Other studies on similarly sized skulls have found a comparably large biological variation (e.g., on human and great ape skulls, Martnez-Abadas et al., 2012; Singh et al., 2012). In contrast to our study, Robinson & Terhune (2017) found evidence that patterns among specimens at the intraspecific level may be obscured by scanner-Microscribe differences and/or between-operator error. However, their result is based on the much smaller species Callicebus (male body mass = 1.02 kg), than the grey seal (male body mass = 170–310 kg). As it generally can be assumed that the effect and relative contribution of measurement error becomes larger as true biological variation becomes lower (Fruciano, 2016; Fruciano et al., 2017; Fruciano et al., 2019), we repeated the same analysis based on only the 17 specimens from the Baltic Sea population, excluding the five specimen from the western North Atlantic population, however, we obtained a similarly large biological variation (Table S2).

In contrast to previous studies comparing 3D surface scanners to other devices in the context of geometric morphometrics data collection (e.g. Sholts et al., 2011; Algee-Hewitt & Wheat, 2016; Fruciano et al., 2017; Robinson & Terhune, 2017; Shearer et al., 2017; Marcy et al., 2018; Giacomini et al., 2019), we used an open-source system (Eiriksson et al., 2016) rather than a commercial scanner. Using such an open-source system has the advantage of having full control over the scanning pipeline. In contrast to using commercial scanners, we could freely decide how to clean the noisy raw scans, and which reconstruction algorithm we wanted to use, thus allowing us to minimize scanning-related measurement error. As this system comes with an increased amount of freedom, an operator had to make some choices with respect to scanning parameters, such as exposure time, or the distance between the scanner and the specimen to scan, even when following a general scanning protocol. Our analysis of the Extended Scanner Dataset revealed that the error due to these scanning-related differences was significant, but smaller than other errors, also within-operator error, which is linked to precision. Our analysis further suggests that 3D digital models, which are based on scans from different operators that followed a general protocol, were only a minor source of total shape variation. We note that even though the distance between scanner and the specimen is linked to measurement error due to optical distortion (an example of presentation error (Fruciano, 2016)), we could not systematically analyse this error source, as parameter choice was at the discretion of the operators.

We note that we could have constructed the Device Comparison Dataset differently by using the datasets where each operator placed the landmarks on the 3D models reconstructed from the scans made by the other operator (datasets scAB1, scAB2, scBA1, scBA2) instead of their own scans (datasets scAA1, scAA2, scBB1, scBB2). By doing so, we noticed a trend of slightly lower repeatabilities when comparing devices by means of pairwise Procrustes ANOVA (Table S3), however, we did not find significant differences in the Nested Procrustes ANOVA analyses (Table S4).

Conclusions and recommendations

In this study, we used a carefully chosen setup, which allowed the investigation of multiple sources of measurement error (within- and between-operator, between-device, and between-scan), and we accounted for the compounding of error between operator and digitizer by having each operator digitizing separate scans.

Our results suggest that the SeeMaLab scanner pipeline is well suited for landmark measurement on grey seal skulls, and that it provides a viable alternative to a Microscribe digitizer for geometric morphometrics at intraspecific level for skulls of similar sizes. On average, we obtained a similar precision with both devices. However, we found evidence that for a few skulls, operators were less consistent in replicating annotation of landmarks on the scanner-based 3D models compared to the digitizer. This was linked to a few difficult landmarks. By omitting difficult landmarks, we found that the scanner pipeline resulted in a higher precision compared to the digitizer, and the problem that an operator was not always consistently placing landmarks was overcome. Thus, we would recommend any user of a scanner pipeline for geometric morphometrics to do a pre-study to identify difficult to place landmarks, and develop a strategy of how to deal with these landmarks (e.g., leave them out, or repeatedly measure them and use an average) in order to obtain a precision as high as possible. We would like to stress that both a 3D surface scanner and a digitizer can be used for geometric morphometrics, and that they both come with their respective strengths and limitations.

Our study further points out that between-operator error, which is one of the largest measurement error sources, is biased and seems to be smaller when using the scanner pipeline than the digitizer. Especially for landmarks placed on intersecting sutures, there is evidence that between-operator error is smaller using the scanner. Moreover, we found systematic differences between devices, mainly driven by landmarks that are not clearly defined. For our data-set of relatively large skulls, where operator and device are not contributing much to total shape variation, it might not be a problem combining data from two operators, or indeed two devices. However, since we observed systematic error between both devices and operators, we would not advise to combine digitizer and scanner data, or data from different operators when the expected biological variation is small without conducting a pre-study and testing the landmarks for repeatability across platforms and operators. For any study where it will be necessary to combine landmark measurements from different operators, the scanner pipeline will be better suited due to the smaller between-operator error.

In contrast to previous studies, which compared the use of 3D surface scanners to other 3D capture modalities or digitizers, we used the SeeMaLab scanner, which is an open-source system developed at the Technical University of Denmark. The use of such a system has the advantage of full control over the process of generating 3D models. The scanning-related error, i.e., the error due to different scanning parameter choices, is very small compared to all other error sources for our data set. Thus, we do not expect the landmark measurements to vary substantially between different scans. Furthermore, in principle, different operators can scan different specimens without introducing significant error, as long as they all follow the same scanning protocol. This, again, underlines the importance of a pre-study to set up a common scanning protocol in order to obtain satisfying 3D models of the specimens of interest. It would be interesting to systematically investigate presentation error due to optical distortion or different reconstruction algorithms and compare it to other error sources in future work. Since the SeeMaLab scanner so far mainly has been used as a research platform, we note that the scanner comes with some practical issues (e.g., profound knowledge required; manual steps; reflection and focus issues at specimen teeth), however, the scanner pipeline could easily be improved by automating parts of the pipeline and implementing quality control and high-dynamic range techniques.

Supplemental Information