Goal-oriented evaluation of species distribution models’ accuracy and precision: True Skill Statistic profile and uncertainty maps
- Published
- Accepted
- Subject Areas
- Biogeography, Conservation Biology, Ecology
- Keywords
- AUC, Bayesian inference, Chelonoidis chilensis, sensitivity, presence-only data, specificity, MaxEnt, TSS
- Copyright
- © 2015 Ruete et al.
- Licence
- This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, reproduction and adaptation in any medium and for any purpose provided that it is properly attributed. For attribution, the original author(s), title, publication source (PeerJ PrePrints) and either DOI or URL of the article must be cited.
- Cite this article
- 2015. Goal-oriented evaluation of species distribution models’ accuracy and precision: True Skill Statistic profile and uncertainty maps. PeerJ PrePrints 3:e1208v1 https://doi.org/10.7287/peerj.preprints.1208v1
Abstract
The use of species distribution models’ (SDM) is limited by its performance in terms of accuracy, precision, or the spatial distribution of model errors. Despite the wide acceptance of some standard statistics used to evaluate SDM, there is currently a strong on-going debate as to their use. The “area under the curve” (AUC) is a popular measure used to evaluate SDMs; however, it does not provide complete information about model accuracy. The maximum True Skill Statistic (TSS) is another statistic that is gaining acceptance. However, evaluations of a model’s accuracy solely based on this statistic may also be misleading. We investigate the use of alternative methods to evaluate the performance of SDMs, to objectively compare among different modelling approaches. We evaluate the performance of SDMs fitted to simulated and real data by contrasting model predictions to additional validation datasets. We propose visualising TSS scores over the whole detection threshold range (TSS profile). We show how models with similarly good performance according to AUC, present very different results and may serve to different purposes. Also, a high maximum TSS may not guarantee accurate predictions and should be accompanied by the threshold where the maximum is reached (t*). We observe that the higher t* the better predicted observations correlate with confirmed observations. Also, SDM predictions should be accompanied with the corresponding uncertainty map to avoid misleading conclusions. Too high or too widely spread uncertainty on such maps would question the overall accuracy of the model. Whether the model is intended to detect all potential observation sites (sensitive model) or to accurately predict where confirmed observations could be found (specific model) sets a different performance targets to be achieved by the model. The approach proposed helps to discern which SDM may best suit the intended goals. Furthermore, the TSS profile helps i) to evaluate the overall performance of SDMs and compare among them, ii) to identify the main source of error, and iii) to select a detection threshold depending on the maps intended use.
Author Comment
This article will be a submission to PeerJ for review, or to another journal.
Supplemental Information
Supporting information
Additional Supporting Information cited in the main text of this article: Fig. S1. Sensitivity and specificity of simulated SDM Fig. S2. Bayesian spatially expanded logistic (BSEL) model uncertainty Appendix S1. Bayesian spatially expanded logistic (BSEL) model and model selection procedure Table S1. Complete list of observations and sources. Table S2. Explanatory variables and BSEL model selection. Table S3. Presence of Chelonoidis chilensis on protected areas in Argentina and Bolivia.