Technical problem (science)
5.3 GUIDELINES FOR INTEGRATING UNCERTAINTY INTO POLICY ADVICE
Two particular strategies to deal with uncertainty have dominated current practices of science for policy: uncertainties are either downplayed to promote radical risk mitigation policies (enforced consensus/overselling certainty), or they are over-emphasised to prevent government intervention in the economy (van der Sluijs et al., 2010). Both promote policy strategies that can result in extreme error-costs for society (van der Sluijs et al., 2010). More sophisticated strategies to deal with uncertainty are emerging (van der Sluijs et al., 2010).
Without pretending to be complete, we discuss three illustrative state-of-the-art approaches to dealing with uncertainty at the science-policy interface: The Guidance approach of the Netherlands Environmental Assessment Agency (A. C. Petersen et al.,
2013); sensitivity auditing (Saltelli, Pereira, van der Sluijs, & Funtowicz, 2013); and the NUSAP notational system (van der Sluijs, 2017).
5.3.1 The Guidance approach of the Netherlands Environmental Assessment Agency
An example of a state-of-the-art approach to uncertainty assessment and communication at the science-policy interface is the Guidance for Uncertainty Assessment and Communication approach to knowledge quality assessment (the Guidance) of the Netherlands Environmental Assessment Agency (Haque et al., 2017;
P. H. M. Janssen, Petersen, van der Sluijs, Risbey, & Ravetz, 2005; A. C. Petersen et al., 2013; van der Sluijs et al., 2008). The Guidance is a comprehensive framework that covers both the substantial and the societal dimensions of uncertainty and quality. It is a proven tool that has been in use for 15 years. The Guidance was developed in 2002 through a partnership between the Netherlands Environmental Assessment Agency and Utrecht University, and has become widely used in that agency. It has reportedly stimulated co-learning processes among scientific advisers and policymakers for a deeper understanding and awareness of uncertainty and its policy implications (A. C.
Petersen et al., 2011).
The Guidance tool adopts a checklist approach, designed to transparently highlight and communicate uncertainties along a scientific assessment process as a way of structuring informed public and policy debate, whether for an Environmental Impact Assessment for a particular project, or for a broader assessment of the body of knowledge used to inform a policy programme (e.g. P. H. M. Janssen et al., 2005; A.
C. Petersen et al., 2013; van der Sluijs et al., 2008). It does not limit its focus to formal, quantitative methods for sensitivity and uncertainty analysis, but extends its scope to the social context of knowledge production, including assumptions and value-loadings. In this way, it systematically guides scientists in an exploration of deeper uncertainties that reside, for instance, in problem framings, expert judgements, and assumed model structures. It provides a heuristic that encourages self-evaluative systematic critical reflection in order to become aware of pitfalls in knowledge production and use. It also provides diagnostic help as to where uncertainty may occur and why (van der Sluijs et al., 2008). The Guidance focuses on six elements of knowledge production and use (Table 3).
Phase in the assessment Key uncertainty and quality issues to critically reflect upon Problem framing (i) Existing frames of the problem, other than that of end users;
(ii) the interconnections with other problems; (iii) any other relevant aspects of the problem not addressed in the research questions; (iv) the role of the study in the policy process; and (v) the way in which the study connects to previous studies on the subject
Stakeholder involvement (i) The relevant stakeholders; (ii) their views, roles, stakes and involvement with respect to the problem; and (iii) the aspects of the problem on which they disagree
Indicator/visualisation selection
(i) Adequate backing for selection; (ii) alternative indicators; and (iii) support for selection in science, society, and politics
Appraisal of knowledge base
(i) The quality that is required; (ii) the current state of knowledge; and (iii) the gap between these two Mapping and assessing
relevant uncertainties
(i) The relative importance of statistical uncertainty, scenario uncertainty and recognised ignorance with respect to the problem at hand; (ii) the uncertainty sources that are most relevant to the problem; and (iii) the consequences of these uncertainties for the conclusions of the study
Communication of uncertainty information
(i) Context of reporting; (ii) robustness and clarity of main messages; (iii) policy implications of uncertainty; (iv) balanced and consistent representation in progressive disclosure of uncertainty information; and (v) traceability and adequate backing
Table 3. Criteria and key issues for knowledge quality in the Guidance (A. C. Petersen et al., 2013).
5.3.2 Sensitivity auditing
Another checklist-based tool that helps to make sense of and gauge the reliability, relevance and legitimacy of model-based inferences is sensitivity auditing (Saltelli
& Funtowicz, 2014; Saltelli et al., 2013). Applying sensitivity auditing implies running through a checklist:
• Rule 1: ‘Check against rhetorical use of mathematical modelling’: are results being over-interpreted? Is the model being used ritually or rhetorically?
• Rule 2: ‘Adopt an ‘assumption hunting’ attitude’: this would focus on unearthing possibly implicit assumptions.
• Rule 3: ‘Detect pseudo-science’: this asks whether uncertainty has been downplayed, as discussed above, in order to present results in a more favourable light. This rule can be referred to as GIGO, from ‘garbage in — garbage out’, or as
‘detect pseudo-science’ (Funtowicz & Ravetz, 1990).
• Rule 4: ‘Find sensitive assumptions before these find you’: this is a reminder that the analysis of sensitivity should be done and made accessible to researchers before publishing results.
• Rule 5: ‘Aim for transparency’: this rule echoes present debates on open data and of the need for a third party to be able to replicate a given analysis; see e.g. the Peer Reviewers’ Openness Initiative (Morey et al., 2016), intended to
discipline authors into providing complete access to the materials used in the preparation of articles, or the San Francisco declaration (American Society for Cell Biology, 2012), as well as Ioannidis’s paper on How to Make More Published Research True (2014).
• Rule 6: ‘Do the right sums’: the analysis should not solve the wrong problem — doing the right sums is more important than doing the sums right. This points to quantitative storytelling, discussed below. In summary, this rule is about asking whether the given quantification is not neglecting important alternatives ways to frame a given example.
• Rule 7: ‘Focus the analysis on the key question answered by the model, exploring holistically the entire space of the assumptions’: this rule is a reminder of good sensitivity analysis practice to run sensitivity analysis globally. Additionally, the object of the analysis should not be the model per se, but the inference of policy being supported by the model. Fragility and volatility are, in fact, not attributes of the model as such, but of the model as used to answer a particular question. An important implication of this rule is that a model cannot be audited for sensitivity once and for all, but needs to be re-audited in the context of each specific application of the model.
This checklist may be seen to pose a burden on the analyst; however, when a scientific analysis is intended to inform an important policy process, it is reasonable to ask that methodological standards be set high (Funtowicz & Ravetz, 1990).
5.3.3 NUSAP
The NUSAP system for uncertainty assessment (Funtowicz & Ravetz, 1990; Refsgaard et al., 2007; van der Sluijs, 2017; van der Sluijs et al., 2005) aims to provide an analysis and diagnosis of uncertainty in science for policy. The basic idea is to qualify quantities by using the five qualifiers of the NUSAP acronym: numeral, unit, spread, assessment, and pedigree.
NUSAP complements quantitative analysis (numeral, unit, spread) with expert judgement of reliability (assessment) and systematic multi-criteria evaluation of the different phases of production of a given knowledge base (pedigree). Pedigree criteria can be proxy representation, empirical basis, methodological rigour, theoretical understanding and degree of validation. Pedigree assessment can be further extended to also address societal dimensions of uncertainty, using criteria that address different types of value ladenness, quality of problem frames, etc.
NUSAP provides insight on two independent uncertainty-related properties expressed in numbers, namely spread and strength. Spread expresses inexactness whereas strength expresses the methodological and epistemological limitations of the underlying knowledge base. The two metrics can be combined in a diagnostic diagram, mapping strength of, for instance, model parameters and sensitivity of model outcome to spread in these model parameters. Neither spread alone nor strength alone is a sufficient measure for quality. Robustness of model output to parameter strength could be good even if parameter strength is low, if the spread in that parameter has
a negligible effect on model outputs. In this situation, our ignorance of the true value of the parameter has no immediate consequences. Alternatively, model outputs can be robust against parameter spread even if its relative contribution to the total spread in the model is high, provided that parameter strength is also high. In the latter case, the uncertainty in the model outcome adequately reflects the inherent irreducible (stochastic) uncertainty in the system represented by the model. Uncertainty then is a property of the modelled system and does not stem from imperfect knowledge on that system. Mapping components of the knowledge base in a diagnostic diagram thus reveals the weakest spots and helps in setting priorities for improvement (Boone et al., 2009; Pye et al., 2018; van der Sluijs et al., 2005).
The strength of NUSAP is its integration of quantitative and qualitative uncertainty. It can be used on different levels of comprehensiveness: from a ‘back of the envelope’
sketch, based on self-elicitation to a comprehensive and sophisticated procedure involving structured, informed, in-depth group discussions on a parameter-by-parameter format. A limitation is that the scoring of pedigree criteria is based on expert judgements. Therefore, outcomes may be sensitive to the selection of experts.
It is thus advisable to involve a diverse group of experts in the pedigree scoring (van der Sluijs, 2017).