3. Mathematical formulation of the identification problemidentification problem
3.1. General setting and assumptions
The implementation of the Bayesian paradigm with respect to statistical inference is given byBayes’ theorem which is closely linked to the concept of conditional probability.
To be able to cast the identification problem in a statistical setting, these concepts are briefly introduced and the necessary notation is fixed. For a detailed introduction to the applied concepts, the reader is referred to general literature on probability theory and statistics such as Billingsley [24], Rohatgi and Saleh [188] or Schay [194].
In the following, a probability space (S,F, P) is employed. It is constituted by a sample space S, a σ-algebra F of events and a probability measure P such that the probability measure P satisfies
(i) 0≤P(A)≤1,∀A∈ F,
3.1. General setting and assumptions
(ii) P(S) = 1, (iii) P(S∞
i=1Ai) =P∞
i=1P(Ai)for the mutually disjoint sets Ai ∈ F.
A fundamental concept in probability theory is that of conditional probability.
Definition 3.1.1 (Conditional probability). On the probability space (S,F, P), with A,B∈ F and P(B)>0, the conditional probabilityP(A|B) is given by
P(A|B) = P(A∩B)
P(B) . (3.8)
By the symmetryP(A∩B) =P(B∩A), this definition directly impliesBayes’ theorem:
P(B|A) = P(A|B)P(B)
P(A) . (3.9)
3.1.1. Random variables
A random variable X is defined as a measurable function on the sample space S such that events ∈ F can be described in terms of sets such as
(a) {s∈ S :X(s) =x} (b) {s∈ S :X(s)≤x} (c) {s∈ S :X(s)∈ B}
for some real value x ∈ Rn and the Borel set B on Rn. These sets are often stated in short as {X= x}, {X≤ x} or {X∈ B}. Throughout this thesis, random variables are indicated by the use of a serif-free front.
The discrete case In discrete sample spaces, the association of a function pX(x) = P({X=x}) =:P(X=x) is straightforward and usually pX is referred to as probability-(mass) function of the random variable X. Since in finite dimensions also (a) represents a proper set, the conditional probability P(X = x|Y = y) is well defined as long as P(Y=y)6= 0.
The continuous case In the continuous case, the probability P(X = x) = 0 due to compliance with the axiom (ii). Anyhow, for a proper setA∈ F, e.g.,A ={X∈ B}, the probabilityP(X∈ B)can be expressed in terms of the Lebesgue integral onRnaccording to
In this setup, the function pX : S → [0,∞[ is referred to as probability density func-tion and must comply with the 3 axioms of probability (i)-(iii). Equivalently, the joint probability density pX,Y(x, y) describes the joint probability P(X∈ B,Y ∈ D) =P(X∈
In this setting, the conditional probability P(X ∈ B|Y = y) is undefined in terms of definition 3.1.1 sinceP(Y=y) = 0. Instead, it is defined as the limit
P(X∈ B|Y=y) := lim
→0+P(X∈ B|Y=]y−, y+]), (3.12) provided it exists [see 188, p. 109]. By inserting this into the definition 3.1.1, it can be shown that a conditional probability density functionpX|Y is given by
pX|Y(x|y) = pX,Y(x, y)
pY(y) (3.13)
such that the probabilityP(X∈ B|Y=y) can be evaluated according to P(X∈ B|Y=y) =
Z
B
pX|Y(x|y)dx. (3.14) The functionpX|Y(x|y)represents a probability density with respect to the random vari-ableX. Seen as a function ofyit is calledlikelihood since it does not represent a density.
Noticing that the conditional density (3.13) can be constructed similarly for pY|X(y|x) and that the joint probability density is symmetric, Bayes’ theorem in terms of densities emerges as a direct consequence as
pX|Y(x|y) = pY|X(y|x)pX(x)
pY(y) . (3.15)
Expected value and variance A random continuous variable X is entirely defined by the associated density pX(x). However, this description is too elusive to identify the characteristic properties of a random variable. Therefore, often summarizing statistics are used to characterize random variables. The most prominent summary is theexpected value
Ex[X] :=
Z
S
xpX(x)dx, (3.16)
provided the integral converges absolutely. It can be seen as a measure of the center of the distribution pX. A measure of the dispersion of the distribution is given by the covariance
COVx[X] :=Ex[(x−Ex[X])(x−Ex[X])]. (3.17) In a multidimensional setting,X={X1, . . . ,Xn}, this information is difficult to visualize.
Results on the variability of a distribution are therefore more conveniently reported in terms of marginal densities pXi(xi). Thus, the variance of Xi is given as the one-dimensional covariance
Vxi[Xi] =Exi[(xi−Exi[Xi])2] = [COVx[X]]ii. (3.18) To quantify this dispersion in the same units as the random variable itself, a specification in terms of the standard deviation
SDx[X] =p
Vx[X] (3.19)
3.1. General setting and assumptions is more convenient. Furthermore, given a random variableXwith probability densitypX and a functionf :X7→f(X)with unknown densitypf(X), the expected valueEf(x)[f(X)]
is given by
Ef(x)[f(X)] = Z
S
f(x)pX(x)dx. (3.20)
Variance Vf(x)[f(X)]and standard deviationSDf(x)[f(X)]are obtained equivalently.
Remark. In the probabilistic literature, it is common to ambiguously use the symbol p to denote different probability distributions such that Bayes’ theorem (3.15) is also stated as
p(x|y) =p(y|x)p(x)
p(y) . (3.21)
Thereby, the specific functional relation of a distribution has to be derived from the argu-ment. Whenever misinterpretation is impossible, this convention is also applied through-out this thesis.
3.1.2. Statistical identification problem
Bayes’ theorem (3.15) provides the basis for the statistical formulation of the identifi-cation problem. To this end, the observations Z ∼ p(Z) and the parameters θ ∼p(θ) are interpreted as random variables with Z ∈ Z and θ ∈ Rnp. The dimension np of the identification problem is thereby introduced to decouple the strict association of the parametersθto the coefficients of the element-wise basis (2.105). This allows for a more flexible interpretation of the parametersθ in terms of a different basis or in the sense of an assembly of different physical parameters. In this setting, the identification problem is given by the posterior density p(θ|Z) which is defined by the application of Bayes’
theorem:
p(θ|Z) = p(Z|θ)p(θ)
p(Z) . (3.22)
In the context of statistical inference, the densities p(Z|θ), p(θ) and p(Z) are often referred to aslikelihood (as a function ofθ),prior and evidence, respectively.
The practical application of (3.22) requires the formulation of a statistical relation θ7→Zsuch that the likelihood functionp(Z|θ) can be defined. Furthermore, knowledge on the statistical universe of the parametersθ must be available in terms of a priorp(θ).
In the limit, this knowledge can also be non-informative. In this case, the data Z must provide enough information such that (3.22) represents a probability density.
In contrast to the classical optimization problem (3.5), the solution of the identification problem is now given as a probability density. This fact directly reveals the ambiguity in the definition of a single optimal solution. In order to define some notion of optimality (see chapter 4) it is often sufficient to observe the proportionality
p(θ|Z)∝p(Z|θ)p(θ). (3.23) The remainder of this chapter is concerned with the specific definitions of the likelihood function p(Z|θ) and the prior-densityp(θ)such that the relation equation (3.23) is well defined. To this end, it is assumed that therandomness in the observations Zis entirely created by noise in the measurement process. Furthermore, it is assumed that this noise is
represented by a normally distributed random variableζ with zero mean and covariance ΣZ:
ζ ∼ N(0,ΣZ). (3.24)
Throughout this thesis, the measurement covariance ΣZ is assumed to be known or to be inferred from the measurement process a priori. Therefore, it represents a user input to the problem usually given in the form ΣZ =σ2I.
Given the specific definition of the measurement noise (3.24), the observation Z is related to the computational model via the additive noise model
Z=C(A(θ)) +ζ. (3.25)
Whenever the compound application F(θ) := (C◦A)(θ) is not of particular interest in a specific context, this relation is often stated more concisely as
Z=F(θ) +ζ. (3.26)
The generic function F :Rnp → Z is also referred to as model (and also computational model) but not to be confused with the parameter-to-state mapping (3.1). Despite the simplified form, it has to be kept in mind that the functional dependence θ 7→ F(θ) is not explicitly available. A rearrangement of (3.26) gives rise to the likelihood in terms of a Gaussian probability density
p(Z|θ) = 1
p(2π)m|ΣZ|exp(−1
2kZ−F(θ)k2Σ−1
Z ). (3.27)
Using the simplification ΣZ =σ2I and a generic similarity measure D:Z × Z → R+0, the likelihood can be written as
p(Z|θ) = 1
p(2πσ2)mexp(− 1
2σ2D(Z, F(θ))). (3.28) In order for this generic formulation to represent a Gaussian probability distribution, the similarity measureD must be induced by a scalar-product in Rm<∞. The presentation throughout this thesis is therefore restricted to a setup defined by
D(Z, F(θ)) :=hZ−F(θ),Z−F(θ)iZ ≡ hfZ(Z−F(θ)), fZ(Z−F(θ))iRm, (3.29) whereby fZ : Z → Rm is seen as a generic function that maps from a potentially continuous spaceZ to some discrete spaceRm. Its existence is implied by proper choices of similarity measures Dand vice versa.
Remark. It is to be mentioned, that the additive noise model (3.26) is not the only one and more possibilities exist [110, 116]. The choice for a specific model has to be made in combination with an experimental setup.
The assumption on the availability of a priori knowledge of parameters of the mea-surement noise is not a necessary assumption and does not affect the validity of Bayes’
theorem (3.23). In fact, from a fully Bayesian perspective noise-parameters such as the