Surrogate Modelling and Uncertainty Quantification in Computational Sciences

(1)

Research Collection

Presentation

Surrogate Modelling and Uncertainty Quantification in Computational Sciences

Author(s):

Sudret, Bruno Publication Date:

2020-08-27 Permanent Link:

https://doi.org/10.3929/ethz-b-000469598

Rights / License:

In Copyright - Non-Commercial Use Permitted

This page was generated automatically upon download from the ETH Zurich Research Collection. For more information please consult the Terms of use.

ETH Library

(2)

Surrogate Modelling and Uncertainty Quantification in Computational Sciences

Bruno Sudret

Chair of Risk, Safety and Uncertainty Quantification, ETH Zurich

(3)

Introduction

Surrogate modelling & UQ Luzern – August 27, 2020 B. Sudret 1 / 40

(4)

What is a computational model?

Complex natural or engineering systems are investigated / designed and assessed usingcomputational models, a.k.asimulators

A computational model combines:

• _Amathematical descriptionof the physical phenomena (governing equations),e.g.mechanics, electromagnetism, fluid dynamics, etc.

divσ+f=0 σ=D·ε

ε= 1

2 ∇u+^T∇u

• Discretization techniqueswhich transform continuous equations into linear algebra problems

• Algorithms tosolvethe discretized equations

(5)

Why do we use computational models?

• _Tobetter understandphysical phenomena,i.e.test theories and assumptions against real-world observations

Model calibration

• _Toanswer “what if?” questions: vary parameters within some ranges and see what happens Parametric study

• To find outimportant parametersthat drive the model predictions

Sensitivity analysis

(6)

Why do we use computational models (in engineering)?

• _Toexplore the design spaceby creating virtual prototypes Model exploration

• _To_optimizethe system’s performance (e.g.minimize its mass while ensuring certain behaviour)

Optimization

• To assess itsrobustnessw.r.t uncertainties in the environmental & usage conditions

Uncertainty quantification / reliability

(7)

What about computational costs?

• Computer power has grown tremendously over the last decades (GigaFlops→TeraFlops→PetaFlops→...)

• Modellers already use the available power fora single run e.g. “virtual universe simulation” by Teyssier et al.:

∼80 hours on 4,000⁺GPU nodes

Piz Daint Super Computer

Cosmic web (Image: J. Stadel)

How to carry out a parametric study / model exploration with:

• Costly simulators

• Complex input/output (nonlinear) behaviour

• High-dimensional input space

Surrogate models

(8)

Outline

Surrogate models

Basics of uncertainty quantification

Polynomial chaos expansions Principle

Computing the coefficients

Applications Subsurface flow

Machine learning benchmarks

(9)

Surrogate models

Input X

Output Y =M(X) Computational model

M

Asurrogate modelM˜ is anapproximationof the original computational modelMwith the following features:

• It is built from alimitedset of runs of the original modelMcalled theexperimental design X =

x⁽ⁱ⁾, i= 1, . . . , n that yield the model responsesY=

y⁽ⁱ⁾=M x⁽ⁱ⁾

, i= 1, . . . , n

• It assumes some regularity of the modelMand some generalfunctional shape

• _{It is}fast to evaluate

(10)

Surrogate models: examples

Name Shape Parameters

Polynomial chaos expansions M(x) =˜ X

α∈A

aαΨα(x) aα

Kriging (a.k.a Gaussian processes) M(x) =˜ β^T·f(x) +Z(x, ω) β, σ_Z²,θ

Support vector machines M(x) =˜

m

X

i=1

aiK(xi,x) +b a, b

Neural networks M(x) =˜ f2(b2+f1(b1+w1·x)·w2) w,b

Low-rank tensor approximations M(x) =˜

R

X

l=1

bl M

Y

i=1

v_l⁽ⁱ⁾(xi)

!

bl, z⁽ⁱ⁾_k,l

(11)

Ingredients for building a surrogate model

• _{Select an}experimental designXthat covers at best the domain of input parameters: Latin hypercube sampling (LHS), low-discrepancy sequences

• Run the computational modelMontoX

• Smartly post-process the data{X,M(X)}through alearning algorithm

Name Learning method

Polynomial chaos expansions sparse grid integration, least-squares, compressive sensing Low-rank tensor approximations alternate least squares

Kriging maximum likelihood, Bayesian inference

Support vector machines quadratic programming

• _Validatethe surrogate model,e.g.estimate a global errorε=E h

M(X)−M(X)˜ ²i

(12)

Wait, isn’t it machine learning?

I. Goodfellow, Y. Bengio, A. Courville,Deep learning, MIT Press (2017)

• Machine learningaims at makingpredictionsby building a model based on data

• Unsupervised learningaims at discovering a hidden structure within unlabelled data x⁽ⁱ⁾, i= 1, . . . , n

• Supervised learningconsiders atraining data set:

X =

(x⁽ⁱ⁾, y⁽ⁱ⁾), i= 1, . . . , n

where:

– x⁽ⁱ⁾’s are theattributes/ features (input space) – y⁽ⁱ⁾’s are thelabels(output space)

(13)

Wait, isn’t it machine learning?

Classification

• _Inclassificationproblems, the labels are discrete,e.g.y⁽ⁱ⁾∈ {−1,1}. The goal is topredict the classof a new pointx

Logistic regression - Support vector machines - (Deep) neural networks

Regression

• _In_regressionproblems, the labels are continuous, sayy⁽ⁱ⁾∈ DY ⊂R^. The goal is topredict the valueyˆ= ˜M(x)for a new pointx

Neural networks - Gaussian process models - Support vector regression

0 5 10 15

−15

−10

−5 0 5 10 15

(14)

Bridging supervised learning and surrogate modelling

Features Supervised learning Surrogate modelling

Computational modelM

7 4

Input spaceX∼fX

7 4

Training data:X ={(xi, yi), i= 1, . . . , n}

4 4

Training data set Experimental design

(big data) (small data)

Prediction goal: for a newx∈ X/ ,y(x)?

m

X

i=1

yiK(xi,x) +b

Validation (resp. cross-validation)

4 4

Validation set Leave-one-out CV

(15)

Advantages of surrogate models

Usage

M(x) ≈ M(x)˜

hours per run seconds for10⁶runs

Advantages

• Non-intrusive methods: based on runs of the computational model

• Suited to high performance computing:

“embarrassingly parallel”

• Similarities withbig data analysis

Challenges

• Need for rigorousvalidation

• Communication: advanced mathematical background

(16)

Outline

Surrogate models

Basics of uncertainty quantification Polynomial chaos expansions Applications

(17)

Global framework for uncertainty quantification

Step A

Model(s) of the system Assessment criteria

Step B

Quantification of sources of uncertainty

Step C

Uncertainty propagation

Random variables Computational model Moments

Probability of failure Response PDF

Step C’

B. Sudret,Uncertainty propagation and sensitivity analysis in mechanical models – contributions to structural reliability and stochastic spectral methods (2007)

(18)

Step B: Quantification of the sources of uncertainty

Goal:represent the uncertain parameters based on theavailable

data and information Probabilistic modelf_X

Experimental data is available

• What is thedistributionof each parameter ?

• What is thedependence structure? Copula theory

0 2 4 6

0 2 4 6 8 10

0 100 200 300 400

Data Normal LN Gamma

?

No data is available: expert judgment

• Engineering knowledge (e.g.reasonable bounds and uniform distributions)

• Statistical arguments and literature (e.g.extreme value distributions for climatic events)

Scarce data + expert information

Bayesian statistics

(19)

Step C: uncertainty propagation

Goal:estimate the uncertainty / variability of thequantities of interest(QoI)Y =M(X)due to the input uncertaintyfX

• Output statistics,i.e.mean, standard deviation, etc.

µY =EX[M(X)]

σY² =EX

(M(X)−µY)²

Mean/std.

deviation

µ σ

• Distributionof the QoI

Response PDF

• Probability of exceedingan admissible thresholdyadm

Pf =P(Y ≥yadm)

Probability of

failure P_f

(20)

Uncertainty propagation using Monte Carlo simulation

Principle

Generatevirtual realizationsof the system usingrandom numbers

• A sample setX={x1, . . . ,xn}is drawn according to the input distributionfX

• For each sample the quantity of interest (resp. performance criterion) is evaluated, say Y={M(x1), . . . ,M(xn)}

• The set of model outputs is used for moments-, distribution-, quantile- or reliability analysis

(21)

Uncertainty propagation using Monte Carlo simulation

• •

• •••

• • • • X1

•• •

•• ••••• X2

• ••

• •

•• • X3

Computational model

•• •Y

••• ••

• •

(22)

Advantages/Drawbacks of Monte Carlo simulation

Advantages

• Universal method: only rely uponsampling random numbers and running repeatedly the computational model

• Sound statistical foundations: convergence whenn→ ∞

• _{Suited to}High Performance Computing:

“embarrassingly parallel”

Drawbacks

• Statistical uncertainty: results are not exactly reproducible when a new analysis is carried out (handled by computingconfidence intervals)

• Low efficiency: convergence rate∝n^−1/2

Monte Carlo for reliability analysis

To computePf= 10^−kwith an accuracy of±10% (coef. of variation of 5%),4·10^k+2runs are required

Need for surrogate models !

(23)

Outline

Surrogate models

Applications

(24)

Polynomial chaos expansions in a nutshell

Ghanem & Spanos (1991); Sudret & Der Kiureghian (2000) Xiu & Karniadakis (2002); Soize & Ghanem (2004)

• _InputXwith given PDFfX(x) =QM

i=1fX_i(xi)(dimX=M)

• _OutputY =M(X)cast as the following polynomial chaos expansion:

Y = X

α∈N^M

yαΨα(X)

where :

• Ψα(X): basisfunctions

• yα: coefficientsto be computed (coordinates)

• _{PCE basis}

Ψα(X),α∈N^M made ofmultivariate orthonormal polynomials Ψα(x)^def=

M

Y

i=1

Ψ⁽ⁱ⁾α_i(xi)

(25)

Multivariate polynomial basis

Univariate polynomials

• For each input variableXi, univariate orthogonal polynomials{P_k⁽ⁱ⁾, k∈N}are built:

D

P_j⁽ⁱ⁾, P_k⁽ⁱ⁾E

= Z

P_j⁽ⁱ⁾(u)P_k⁽ⁱ⁾(u)fX_i(u)du= γ_j⁽ⁱ⁾δjk

e.g.,Legendre polynomialsifXi∼ U(−1,1),Hermite polynomialsifXi∼ N(0,1)

• Normalization:Ψ⁽ⁱ⁾_j =P_j⁽ⁱ⁾/ q

γ_j⁽ⁱ⁾ i= 1, . . . , M, j∈N Tensor product construction

Ψα(x)^def=

M

Y

i=1

Ψ⁽ⁱ⁾α_i(xi) E[Ψα(X)Ψβ(X)] =δαβ

whereα= (α1, . . . , αM)are multi-indices (partial degree in each dimension)

(26)

Multivariate polynomial basisM = 2

α= [3,3] Ψ(3,3)(x) = ˜P3(x1)·He˜3(x2)

• X1∼ U(−1,1): Legendrepolynomials

• X2∼ N(0,1): Hermitepolynomials

(27)

Outline

Surrogate models

Applications

(28)

Computing the coefficients by least-square minimization

Isukapalli (1999); Berveiller, Sudret & Lemaire (2006)

Principle

The exact (infinite) series expansion is considered as the sum of atruncated seriesand aresidual:

Y =M(X) =X

α∈A

yαΨα(X) +εP ≡Y^TΨ(X) +εP(X)

where : Y={yα,α∈ A} ≡ {y0, . . . , yP−1} (P unknown coefficients) Ψ(x) ={Ψ0(x), . . . ,ΨP−1(x)}

Least-square minimization

The unknown coefficients are estimated by minimizing themean square residual error:

Yˆ= arg minE h

Y^TΨ(X)− M(X)2i

(29)

Discrete (ordinary) least-square minimization

Yˆ= arg min

Y∈R^P

1 n

n

X

i=1

Y^TΨ(x⁽ⁱ⁾)− M(x⁽ⁱ⁾)² Procedure

• Select a truncation scheme,e.g.A^M,p=

α∈N^M : |α|1≤p

• _{Select an}experimental designand evaluate the model response M=

M(x⁽¹⁾), . . . ,M(x⁽ⁿ⁾) ^T

• Compute the experimental matrix Aij= Ψj x⁽ⁱ⁾

i= 1, . . . , n; j= 0, . . . , P−1

• Solve the resultinglinear system

Yˆ= (A^TA)⁻¹A^TM Simple is beautiful !

(30)

Validation: error estimators

• In least-squares analysis, thegeneralization erroris defined as:

Egen=E h

M(X)− M^PC(X)²i

M^PC(X) =X

α∈A

yαΨα(X)

Leave-one-out cross validation

• From statistical learning theory,model validationshall be carried out usingindependent data

• LOO cross-validation for PCE emulates it using all data at once ELOO= 1

n

X

i=1

M(x⁽ⁱ⁾)− M^{P C}(x⁽ⁱ⁾) 1−hi

2

wherehiis thei-th diagonal term of matrixA(A^TA)⁻¹A^T,Aij= Ψj(x⁽ⁱ⁾)

x⁽ⁱ⁾

(31)

Outline

Surrogate models

Basics of uncertainty quantification Polynomial chaos expansions

(32)

Example: sensitivity analysis in hydrogeology

Source: http://www.futura-sciences.com/

Source: http://lexpansion.lexpress.fr/

• When assessing anuclear waste repository, the Mean Lifetime Expectancy MLE(x) is the time required for a molecule of water at point x to get out of the boundaries of the system

• Computational models have numerous input parameters (in each geological layer) that aredifficult to measure, and that show scattering

(33)

Geological model Joint work with University of Neuchâtel

Deman, Konakli, Sudret, Kerrou, Perrochet & Benabderrahmane, Reliab. Eng. Sys. Safety (2016)

• Two-dimensional idealized modelof the Paris Basin (25 km long / 1,040 m depth) with5×5m mesh (10⁶elements)

• Steady-state flowsimulation with Dirichlet boundary conditions:

∇ ·(K· ∇H) = 0

• 15 homogeneous layerswith uncertainties in:

– Porosity (resp. hydraulic conductivity)

– Anisotropy of the layer properties (inc. dispersivity) – Boundary conditions (hydraulic gradients)

78 input parameters

(34)

Sensitivity analysis

10⁻¹² 10⁻¹⁰ 10⁻⁸ 10⁻⁶ 10⁻⁴ 10⁻²

T D1 D2 D3 D4 C1 C2 C3ab L1a L1b L2a L2b L2c K1K2 K3

Kbx[m/s]

Geometry of the layers Conductivity of the layers

Question

What are the parameters (out of 78) whose uncertainty drives the uncertainty of the prediction of the mean life-time expectancy?

(35)

Sensitivity analysis: results

Technique:Sobol’indicescomputed from polynomial chaos expansions

0.01 0.2 0.4 0.6 0.8

φ^D4 φ^C3ab φ^L1b φ^L1a φ^C1 ∇H²φ^L2a φ^D1 A^D4_K A^C3ab_a Total Sobol’ Indices

SToti

Parameter P

jS_j φ(resp.K_x) 0.8664

A_K 0.0088

θ 0.0029

α_L 0.0076

A_α 0.0000

∇H 0.0057

Conclusions

• _Only200 model runsallow us to detect the 10 important parameters out of 78

• Uncertainty in the porosity/conductivity of5 layersexplain 86% of the variability

• Small interactions between parameters detected

(36)

Bonus: univariate effects

Theunivariate effectsof each variable are obtained as a straightforward post-processing of the PCE

Mi(xi)^def=E[M(X)|Xi=xi], i= 1, . . . , M

0.05 0.1 0.15

−5 0 5

x 10⁴

φ^D4 MPCE i

0.08 0.1 0.12

−5 0 5

x 10⁴

φ^C3ab

0.14 0.16 0.18

−5 0 5

x 10⁴

φ^L1b

0.1 0.15 0.2

−5 0 5

x 10⁴

φ^L1a MPCE i

0.02 0.04 0.06

−5 0 5

x 10⁴

φ^C1

(37)

Outline

Surrogate models

Basics of uncertainty quantification Polynomial chaos expansions

(38)

Combined cycle power plant (CCPP)

Data set UC Irvine Machine Learning Repository

• 9,568 data points

• 4 features:

- TemperatureT∈[1.81,37.11]^◦C

- Exhaust vacuum in the steam turbineV ∈[25.36,81.56]cm Hg - Ambient pressureP∈[992.89,1033.30]mB

- Relative humidity in the gas turbineRH∈[25.56−100.16]%

• _Output:net hourly electrical energy outputEP ∈[420.26,495.76]MW

Reference approach Tüfekci, P. (2014),Int. J. Elec. Power & Energy Systems

• 13 ML techniques includingregression trees, ANNandSVR

• 10 pairs of training / validation sets of size 4,784

• Best approach:bagging reduced error pruning (BREP) regression tree

(39)

CCPP: Training data (X-space)

-10 0 10 20 30 40 50

0 200 400 600 800 1000

20 30 40 50 60 70 80

0 500 1000 1500 2000

98099010001010102010301040 0

200 400 600 800 1000 1200 1400

0 20 40 60 80 100 120

0 200 400 600 800 1000

(40)

CCPP: Results

Relative mean absolute error M AE= 1 n_val

X

(x,y)∈Xval

|y− M^PC(x)|

MAE min. MAE mean-min rMAE (%)

aPCEonX 3.11±0.03 3.05 0.06 0.68±0.007

BREP-NN^† 3.22±n.a. 2.82 0.40 n.a.

†Tüfekciet al.(2014)

420 440 460 480 500

e(MWh)

0 0.01 0.02 0.03

fE(e)

Estimated PDF of the energy produced by the CCPP:

• Histogram of raw data

• PDF obtained by PCE (10 diff.

training sets) for input dependencies modelled by C-vines

(41)

Airfoil

Data set UC Irvine Machine Learning Repository

• 750 training points, 750 validation points

• 41 features:

– Frequency, in Hertz – Angle of attack, in degrees – Chord length, in meters

– Free-stream velocity, in meters per second.

– Suction side displacement thickness, in meters – 36 noise variables (standard normal)

• _Output:Scaled sound pressure level, in decibels

Reference approach K. Kandasamy & Y. Yu, ICML16 Proc. of the 33rd Int. Conf. on Machine Learning (2016)

• Sparse LASSO regression (SALSA)

• Beats 13 other regression models, incl. neural networks

(42)

Airfoil: Results

(Relative) mean absolute error (MAE)

MAE (dB) rMAE (%) aPCEonX 3.04±0.07 2.4±0.06 SALSA^† 3.81±0.06 3.1±0.04

†Kandasamy & Yu (2016)

(43)

Conclusions

• Surrogate modelsare unavoidable when dealing with costly computational models for uncertainty quantification, sensitivity analysis or optimization

• Depending on the analysis, specific surrogates are most suitable,e.g.polynomial chaos expansions for distribution- and sensitivity analysis,Krigingfor reliability analysis

• All these techniques arenon-intrusive: they rely on experimental designs, the size of which is a user’s choice

• _{They are}_versatile,general-purposeandfield-independent

• All the presented algorithms are available in the general-purpose uncertainty quantification software UQLab

(44)

www.uqlab.com

(45)

UQLab features

(46)

UQLab: The Uncertainty Quantification Software http://www.uqlab.com

• ETH license:

+ free access to academia + yearly fee for non-academic usage

• 2,900+ registered users

• 1,280 active users from 87 countries

• About 37% license renewal after one year

Country # Users United States 493

China 365

France 301

Switzerland 238

Germany 221

United Kingdom 134

Italy 110

Brazil 96

India 88

Canada 77

As of August 24, 2020

(47)

(48)

UQWorld: the community of UQ https://uqworld.org/

(49)

Questions ?

Chair of Risk, Safety & Uncertainty Quantification www.rsuq.ethz.ch

The Uncertainty Quantification Software

www.uqlab.com

Thank you very much for your attention !