Verification and validation

particular computer simulation. In the example above, the phenomenon sin-gled out is a satellite whose orbital eccentricity describes the spikes in Figure 3.3. Only particular computer simulations can be implemented on the physical machine for rendering results.

With these facts in mind, I postulate the following working conceptualization of a computer simulation:

A computer simulation consists of two elements: a general computer simulation, which is the implementation of a simulation model on a digi-tal computer ready to be executed. The simulation model represents the general patterns of behavior of a target system, which can be ‘empirical,’

‘theoretical,’ or ‘data systems.’ A general computer simulation produces results when its initial and boundary conditions are filled out, creating in this way a particular computer simulation. A particular computer sim-ulation produces results by reckoning using stepwise ‘analytic methods,’

‘numerical analysis,’ or ‘stochastic techniques.’⁵⁴

Unlike other philosophical literature, I do not need to postulate a workingdefinition for computer simulations since the above conceptualization is enough for showing their epistemic power.

Let me say that this working conceptualization is essentially an analytic step towards the general notion of ‘computer simulation,’ ‘computer simulation results,’

and similar terminology found in current literature, only necessary for my discussion of explanation. Thus, I see no need to change the terminology as long as the context is clear. In this vein, and unless the context requires, I will continue making use of the same terminology as found in current literature. It is not until Chapter 5, when I address the epistemic power of computer simulations, that I need these new terms. The reason for introducing new terminology lies in the differences between asimulation model and a general computer simulation which must be pointed out, such as the fact that the former is receptive to formal verification and validation, whereas the latter is not. Similarly, the notion of particular computer simulation emphasizes the fact that the initial and boundary conditions of the general computer simulation have been fulfilled, singling out one concrete simulation.

In order to measure the correctness of a model one must have either accurate benchmarks or reference values against which to compare the model, or formal meth-ods for confirming the implementation of the model. Benchmarking, according to William Oberkampf and Christopher Roy, is the technique used for measuring the performance of a system, frequently by running a number of standard tests and trials against it. Invalidation,this is fairly straightforward since benchmarkings are high-quality experimental measurements of systems responsible for the quantities of interest.⁵⁶ In verifaction, however, formal methods are the focus for confirming the correct implementation of the model as a computer simulation. In validation, then, the relationship between computation and the empirical world (e.g., experimental data obtained by measuring and observing methods) is the issue,⁵⁷ whereas in veri-fication, the relationship of the simulation to the empirical world is not necessarily at stake.

Now, this way of depicting verification and validation is simply too general. The scientific and computational communities offer a diversity of definitions, depending on the particularities of the systems under study.⁵⁸ The philosophical community, on the other hand, offers a rather sloppy and inexact account of verification and validation. Eric Winsberg, for instance, neglects the formal aspect of verification (see below: code verification). According to him “Verification, [...] is the process of determining whether or not the output of the simulation approximates the true solutions to the differential equations of the original model. Validation, on the other hand, is the process of determining whether or not the chosen model is a good enough representation of the real-world system for the purpose of the simulation”

(Winsberg, 2010, 19-20). Morrison, alternatively, downplays the need for verification and claims that validation is the crucial method for assessing the confidence of the computer simulation results.⁵⁹

Here following, I base my findings only on scientific literature (specifically, com-puter science). Here are two definitions that are largely accepted:

Verification: the process of determining that a computational model accu-rately represents the underlying mathematical model and its solution.

Validation: the process of determining the degree to which a model is an accurate representation of the real world from the perspective of the intended uses of the model. (Oberkampf et al., 2003)

Let us now discuss them separately.

3.3.2.1 Verification

Verificationcan be divided into two approaches: code verification andcalculation verification. The purpose of making such a distinction is to categorize the set of methods for the assessment of the simulation model. Let us see them in turn.

Code verification: the process of determining that the numerical algorithms are correctly implemented in the computer code and of identifying errors in the software. (Oberkampf et al., 2003, 32)

Code verification, then, is a method guided by theoretical and deductive principles.⁶⁰ As mentioned in our previous discussion on formal methods, there is a host of methods for determining that the algorithm has been correctly implemented. A short list includes measured errors specified and implemented as part of the computer software, potential system halting, and accurate representation of the specification by the algorithm, among others.

On the other hand, calculation verification⁶¹ is defined as the method that pre-vents three kinds of errors: human error in the preparation of the code, human error in the analysis of the results, and numerical errors resulting from computing the discretized solution of the simulation model. A definition for calculation verification is the following:

Calculation verification: the process of determining the correctness of the in-put data, the numerical accuracy of the solution obtained, and the correctness of the output data for a particular simulation. (Oberkampf et al., 2003, 34)

Calculation verification is the empirical side of verification. It is based on the com-parison between the results of the simulation against highly accurate solutions of the scientific model. In a sense, calculation verification is similar to validation as-sessment insofar as both compare estimated results with correct results. It most commonly controls spatial and temporal convergence rates, iterative convergence, independence of solutions to coordinate transformations, and similar processes.⁶² 3.3.2.2 Validation

The process ofvalidation (also known astesting) presupposes an inductive way to demonstrate that the results of the simulation represent, in an accurate and precise way, the results of the implemented scientific model. William Oberkampf and Timothy Trucano highlight three key aspects of validation:

i) quantification of the accuracy of the computational model by comparing its responses with experimentally measured responses,

ii) interpolation or extrapolation of the computational model to conditions corresponding to the intended use of the model, and

iii) determination if the estimated accuracy of the computational model, for the conditions of the intended use, satisfies the accuracy requirements speci-fied.(Oberkampf and Trucano, 2008, 724)

The philosopher of science must expect that validation, as an inductive method, faces typical problems of induction. The general problem is that this method only allows validation of a model up to a certain point, and therefore complete validation is absolutely impossible due to the large number of comparisons needed (not to mention the improbability of having all the possible results at hand). Thus, validation is a method for assisting in the detection of errors, but not designed for detecting misrepresentations of the model.⁶³

Besides, we need to keep in mind that validation depends on the capacity of comparing computer simulation results with empirical results; this dual relation between results requires the presence of both, excluding those computer simulations from the analysis for which there is no empirical data counterpart. In this sense, validation is only a suitable concept for those cases in which a computer simulation is representing an actual system, and not a possible or conceivable system.

Figure 3.5 shows in a flow diagram how verification (both code verification and calculation verification) and validation methods are put into effect on standard sci-entific practice.

Theconceptual model here is the product of analyzing and observing the physical system of interest (i.e., what we called the scientific model). In key applications of computational physics (such as computational fluid dynamics, computational solid mechanics, structural dynamics, shock wave physics, and computational chemistry), the conceptual model is dominated by the PDEs used for representing physical quantities.

Two types of model can be identified: a mathematical model, from which the computational or simulation model is created, and aphysical model which, for sim-plicity, we shall identify with an experiment. The computational model, finally, is an operational computer program that implements the conceptual model.

The figure also shows that code verification deals with the fidelity between the conceptual model and the mathematical model, whereas calculation verification deals with the agreement between the simulation results and the expected results of the computational model. Validation, on the other hand, is a quantitative ade-quacy between the solutions of the simulation and the experimental measurements or observations. This adequacy can be determined by a comparison that provides

Figure 3.5: Phases of modeling and simula-tion, and the role of verification and valida-tion (ASME, 2006, 5).

Verifications and validations are, to-gether with the host of methods dis-cusses in Section 3.3, two fundamental methods for asserting the reliability of results of computer simulations. As we shall argue in the next chapter, reliabil-ity of results supports the claim that we

‘know the results of the simulation,’ i.e., they are correct solutions of the model (Chapter 4).

A final comment on verification and validation: although neither software nor hardware can be fully verified or val-idated, researchers are still developing methods that reduce the possibility of errors and increases the credibility of the model. This interest from the scientific community can only mean that both methods are necessary and required for dependable computer simulations. The literature here mentioned can work as a starting point for future studies. Our sole purpose was to briefly discuss verifica-tion and validaverifica-tion as part of the general methodology of computer simulaverifica-tions, the subject of the present chapter.