Multi-parameter models in systems biology are typically ‘sloppy’: some parameters or combinations of parameters may be hard to estimate from data, whereas others are not. One might expect that parameter uncertainty automatically leads to uncertain predictions, but this is not the case. We illustrate this by showing that the prediction uncertainty of each of six sloppy models varies enormously among different predictions. Statistical approximations of parameter uncertainty may lead to dramatic errors in prediction uncertainty estimation. We argue that prediction uncertainty assessment must therefore be performed on a per-prediction basis using a full computational uncertainty analysis. In practice this is feasible by providing a model with a sample or ensemble representing the distribution of its parameters. Within a Bayesian framework, such a sample may be generated by a Markov Chain Monte Carlo (MCMC) algorithm that infers the parameter distribution based on experimental data. Matlab code for generating the sample (with the Differential Evolution Markov Chain sampler) and the subsequent uncertainty analysis using such a sample, is supplied as

By combining experiments and mathematical model analysis, systems biology tries to unravel the key mechanisms behind biological phenomena. This has led to a steadily growing number of experiment-driven modeling techniques (

A standard procedure for parameter estimation is via a collective fit, i.e., estimating the values of the unknown parameters simultaneously by fitting the model to time series data, resulting in a calibrated model. This approach has to cope with several obstacles, such as measurement errors, biological variation, and limited amounts of data. Another often met problem is that different parameters may have correlated effects on the measured dynamics leading to highly uncertain or even unidentifiable parameter estimates (

In this paper we fit an illustrative model and a diverse set of six systems biology models from the BioModels database (_{0.95} which has the nice property that _{0.95} < 1 indicates tight predictions, whereas _{0.95} ≥ 1 implies uncertainty in the dynamics that is likely to obscure biological interpretation. Using this quantifier, we show that some models allow more accurate predictions than others, but, more importantly, that the uncertainty of the predictions may greatly vary within one model. We argue that prediction uncertainty assessment must therefore be performed on a per-prediction basis using a full computational uncertainty analysis (

The model dynamics are fitted to noisy time series data, typically by an MCMC algorithm, resulting in a sample of parameter values representing their posterior distribution. Typical applications of this distribution are the computation of a credible region of parameters (showing parameter uncertainty) and full computational uncertainty analysis of predictions.

Within the Bayesian framework (

It is useful at this point to compare the Bayesian approach with the frequentist approach of prediction profile likelihood (PPL) (

Parameter uncertainty and prediction uncertainty play an important role also in hydrology (

In this section we present the methodology to sample the probability distribution of the parameters, and to define the uncertainty of predicted dynamics. To illustrate how this works in practice, we have implemented the used sampling algorithm together with a prediction uncertainty computation algorithm in Matlab^{®} software, together with a sample of the parameters (see the

We consider models that are formulated in terms of differential equations and have the following form: ^{p}, with initial condition ^{m} is a positive function

We adopt the Bayesian framework, in which a prior distribution of the parameters is combined with data to form the posterior distribution _{d}_{d}_{d}_{d}_{i}_{d}_{i}_{i}^{2} a measure of the fitting error: ^{PML}_{k}_{k}_{k}

Since we want to compare the uncertainty in predictions of time courses over different time intervals and for different models, we need a quantifier that is independent from model specific issues, such as the number, and the dimensions of the variables. Also, biologists are generally interested in percentage difference rather than absolute difference, so the quantifier should be independent of the typical order of magnitude. To that end we introduce a measure for prediction uncertainty that satisfies these requirements, but we do not claim that this quantifier is unique. However, our present choice has the advantage that it allows a nice interpretation as will be explained below. We first define a measure for the prediction uncertainty of one component of the output and then average the outcomes over the components.

For a predicted output _{p}_{p}_{p}^{PML}_{p,i} the _{p}

If {_{α}, the _{α} is therefore the deviation of a prediction relative to the penalized maximum likelihood prediction, at confidence level

The algorithm for estimating prediction uncertainty naturally falls apart in two sub-algorithms. Part I deals with estimating parameters by exploiting the prior knowledge, the model, and the data. This first part yields the sample of parameter values representing their posterior distribution. Part II is the focus of this paper, and performs the full computational uncertainty analysis by taking as input the sample of parameter values from the first part and by calculating the prediction for each member of this sample. Part I is computationally more intensive than Part II. Because prospective users of the model need to carry out only Part II, it is essential that the sample of parameter values obtained in Part I is stored and made available. In full:

Part I: Estimation of the posterior parameter distribution

To calculate the posterior

A collection of solutions {

Part II: Computational uncertainty analysis

_{α} is approximated by taking the largest

For visualization of the uncertainty in the predicted systems dynamics, for each

A structural difference between _{b}(_{p}_{b}(_{p}^{PML}

To demonstrate what kind of effects can be expected when studying prediction uncertainty of models with parameter uncertainty, we use a model example that represents a highly simplified case of self-regulation of gene transcription: _{1} and _{2}, respectively. They are to be estimated from time series data, for which we used simulated data at ten time points (_{1} and _{2} by generating 1000 draws from their distribution using MCMC sampling. This yielded a sample of 1000 draws. From this sample we derived a 95% credible region of _{1} and _{2} (

(A) The model dynamics fitted to the time series data. The 95% uncertainty region is displayed in yellow, the error bars denote the standard deviation of the measurement noise. (B) Parameter uncertainty represented by the 95% credible region, containing the true parameter vector (red *). (C) Predicted dynamics for different initial concentrations (Predictions 1 and 2), and for a 7 fold amplitude increase of the sinusoidal input (Prediction 3). (D) Predicted dynamics via local sensitivities (linearized covariance analysis). The uncertainty of Prediction 3 is plotted in a different shade and includes the uncertainty region of Prediction 1.

The data give much more information about _{2} than about _{1} (_{2}, but not sensitive to changes in _{1}; the model thus shows sloppiness. The reason is not hard to understand: since we have used a high initial value, most of the _{1} = 10, so that the parameter _{1} has little influence on the dynamics (see

The prediction sets Prediction 1 and Prediction 2 (^{4}), respectively. The uncertainty in prediction increases in time for Prediction 1 and is much higher than that for Prediction 2. The predicted dynamics is thus highly sensitive to parameter uncertainty for Prediction 1, but fairly insensitive for Prediction 2. Based on Predictions 1 and 2 one might guess that the uncertainty is related to the magnitude of

We also estimated prediction uncertainty via a local sensitivity analysis, namely linearized covariance analysis (LCA, see the

This simple example illustrates that prediction uncertainty depends on the type of prediction and is hard to foresee intuitively. We cannot evaluate the prediction uncertainty on the basis of some simple guidelines after only inspecting the credible region of the parameters. Because of the nonlinearities in the system, the credible region is untruthful for evaluating predictions and their uncertainty (

It is convenient to quantify prediction uncertainty, that is, the variability among predicted time courses. We do so on the basis of the quantity _{0.95}. _{0.95} values smaller than 1 indicate tight predictions. For example, the _{0.95} values for Predictions 1, 2 and 3 in

Next, we study prediction uncertainty for six models from the BioModels database (_{0.95}. Perturbations consisted of artificially increasing and decreasing each parameter in turn by a factor of 100 (different factors yielded similar results, see _{0.95} values.

For the six models the prediction uncertainty is displayed in terms of the _{0.95} quantifiers (_{i}_{d,i} per data point _{0.95} distribution ranges considerably differ per model. For example, the Tyson model gives mostly precise predictions, and the Laub model gives mostly uncertain predictions. Next, the accuracy of the data was increased artificially by decreasing the noise with a factor 10, and observed the effect on the _{0.95} distribution ranges (_{0.95} values are reduced, but not in a very systematic way. For the Vilar model, for example, all predictions have become precise, except for one. More importantly, for all six models the _{0.95} values scatter enormously, in ranges that span 2 to 4 orders of magnitude. There is no clear connection between the number of parameters and the maximum _{0.95} values or the size of the ranges. By consequence, prediction uncertainty is not a characteristic of a calibrated model, but must be evaluated on a case-by-case basis.

(A) Uncertainties of predictions of six different models, expressed in terms of _{0.95}. A blue * denotes a tight prediction (

By introducing the quantifier _{0.95} for prediction uncertainty we were able to show that prediction uncertainty is typically not a feature of a calibrated model, but must be calculated on a per-prediction basis using a full computational uncertainty analysis. The adjective

The value of _{0.95} is interpretable. If _{0.95} ≥ 1, the uncertainty is so high that it may obscure biological interpretation of a model prediction, whereas this is unlikely if _{0.95} < 1. When the uncertainty indeed causes predictions to be biologically ambiguous, _{0.95} could be incorporated as a design criterion in experimental design, i.e., additional experiments can be designed or selected to minimize _{0.95} (

We adopted the Bayesian framework in our analysis. This framework has the advantage that in the collective fits prior information on the parameters can be incorporated, such as from general biological knowledge (

Of course there are computational issues in the Bayesian approach which increase with the number of the parameters and the complexity of the model.

Systems biology models may be computationally expensive to evaluate, making standard MCMC algorithms impractical as they require many evaluations. For our class of dynamic computational models the likelihood of each sample is based on the time integration of the model, which can be very demanding for large or stiff models. To confine sampling and integration costs, model order reduction techniques may be of help to speed up analysis (see

An alternative approach is to locally approximate the likelihood function using parameter sensitivities. Applications in systems biology of the latter idea include a gene network model in a Drosophila embryo (

In this paper the analysis is carried out with noiseless data. The Hessian matrix has full rank for each model, so all models are theoretically locally identifiable in the neighbourhood of the penalized maximum likelihood parameter. However, in practice the PML parameter might be hard to find or be non-unique due to noisy data. An alternative is to define

For biologists it is not always essential to obtain quantitative predictions and a reviewer asked about the uncertainty analysis of qualitative predictions. For this, the first step is to convert any quantitative predicted time course into a qualitative prediction, such as the statement that a certain metabolite or flux is going up, when a gene is over-expressed. Once this step has been decided upon, the prediction for any particular set of parameter values in the posterior sample is then either a simple

Our survey of a variety of models shows that prediction uncertainty is hard to predict. It turns out that it is practically impossible to establish the predictive power of a model without a full computational uncertainty analysis of each individual prediction. In practice this has strong consequences for the way models are transferred via the literature and databases. We conclude that publication of a model in the open literature or a database should not only involve the listing of the model equations, the parameter values with or without confidence intervals or parameter sensitivities, but should also include a sample of, say, 1000 draws representing the full (posterior) distribution of the parameters. Only then prospective users are able to reliably perform a full computational uncertainty analysis if they intend to use the model for prediction purposes. To this end, a software package is made available online, including an exemplary parameter sample. In addition we encourage to include experimental conditions, model assumptions, and used prior knowledge in order to make the sample reproducible. This will decrease the chances of error propagation, for example due to programming errors.

Code that was used to generate the computational results in the main text with samples of 1000 parameters sets of each calibrated model.

Code for one model, meant as user-friendly illustration of our method.

This pdf contains further detailed information on the main article.

The authors would like to thank the reviewers for their comments and W Kruijer, H van der Voet, S Schnabel, and E Boer for useful discussions.

Cajo ter Braak is an Academic Editor for PeerJ. Simon van Mourik and Jaap Molenaar are co-financed by the Netherlands Consortium for Systems Biology.