Numerical Analysis and Complexity of Stochastic Processes

18th Jyv¨askyl¨a Summer School

Klaus Ritter

TU Darmstadt, Germany

August 2008

Outline

I. Introductory Example: Reconstruction of a Brownian Motion Stochastic Processes

Brownian Motion

Reconstruction of a Brownian Motion

Outline

I. Introductory Example: Reconstruction of a Brownian Motion Stochastic Processes

Brownian Motion

Reconstruction of a Brownian Motion

Stochastic Processes

Stochastic process (real-valued): a family X = ( X ( t )) _{t ∈D} of random variables

X ( t ) = X ( t , · ) : Ω → R on a probability space (Ω, A , P ).

Paths (trajectories) of X : for fixed ω ∈ Ω

D → R , t 7→ X ( t , ω ).

In these lectures

D = [0, 1] ^d

with d ∈ N , and X has (at least) continuous paths, i.e.,

∀ ω ∈ Ω : X ( · , ω) ∈ C ( D ).

Then X defines a continuous random function X : Ω → C (D ), ω 7→ X ( · , ω ).

Terminology, if d > 1: X random field.

X second-order process if

∀ t ∈ D : E( X ² ( t )) < ∞ .

In this case, the mean and the covariance kernel, m : D → R , K : D × D → R , of X are given by

m ( t ) = E( X ( t )), K (s , t ) = E "

(X (s ) − m(s )) · (X (t ) − m(t )) .

Clearly "

X ( t ₁ ), . . . , X ( t _n )

has mean vector ( m ( t _i )) _1≤ i ≤ n and

covariance matrix (K (t _i , t _j )) _{1≤i ,j ≤n} .

X Gaussian process if

"

X ( t ₁ ), . . . , X ( t _n )

normally distributed

for every n ∈ N and all t ₁ , . . . , t _n ∈ D .

Measures on function spaces: Consider the sup-norm on C (D ) and the respective Borel-σ -algebra B . Then

P _X ( B ) = P ( { X ∈ B } ), B ∈ B ,

is well-defined. Clearly P _X is a probability measure on ( C ( D ), B ), called the distribution of X .

Conversely, for any probability measure µ on ( C ( D ), B ), X ( t , f ) = f ( t )

defines a stochastic process on (C (D ), B , µ), called the canonical

process. Clearly µ X = µ.

Clearly

P _X = P _Y ⇒ m _X = m _Y ∧ K _X = K _Y .

For Gaussian processes X and Y we have ’ ⇔ ‘.

Outline

I. Introductory Example: Reconstruction of a Brownian Motion Stochastic Processes

Brownian Motion

Reconstruction of a Brownian Motion

Brownian Motion

In the sequel

D = [0, 1].

X one-dimensional Brownian motion if (i) X (0) = 0,

(ii) X ( t ₁ ) − X ( t ₀ ), . . . , X ( t _n ) − X ( t _n−1 ) independent for every n ∈ N and all 0 = t ₀ < · · · < t _n ≤ 1,

(iii) X ( t ) − X ( s ) ∼ N (0, t − s ) for 0 ≤ s < t ,

(iv) X has continuous paths.

Existence: Let Z ₁ , . . . , Z _n iid., E( Z ₁ ) = 0, Var( Z ₁ ) = 1, X _n ( i / n ) =

i

X

j =1

√ 1

n · Z _j , i = 0, . . . , n , and use piecewise linear interpolation. Then

P _X

−−−−→ ^weakly

(n→∞) w .

The canonical process on ( C ( D ), B , w ) is a Brownian motion, and w is called the Wiener measure. Compare: central limit theorem.

Simulation: Use random numbers to simulate realizations of

Z ₁ , . . . , Z _n .

Simulations with

P ( Z ₁ = ± 1) = 1/2 and n = 5, 50, 500.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

Simulations with Z ₁ ∼ N (0, 1) and n = 800.

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−0.5 0 0.5 1 1.5

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.5 0 0.5 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1.5

−1

−0.5 0 0.5

t

X(t)

Uniqueness: P _X = w for every Brownian motion X . Applications: physics, finance, analysis, . . .

Lemma Every Brownian motion is Gaussian with zero mean and covariance kernel

K (s , t ) = min(s , t ).

Proof By (i)–(iii), X is Gaussian. Moreover,

m ( t ) = E( X ( t )) = E( X ( t ) − X (0)) = 0 and

K (t , t ) = E(X ² (t )) = E((X (t ) − X (0)) ² ) = t . If s ≤ t then

2 "

k -dimensional Brownian motion X ( t ) = "

X ⁽¹⁾ ( t ), . . . , X ^{( k )} ( t )

with independent one-dimensional Brownian motions

X ⁽¹⁾ , . . . , X ^{(k )} .

Simulation with P ( Z ₁ = ± 1) = 1/2 and n = 50 for both components.

−1 −0.5 0 0.5 1

−1

−0.5 0 0.5 1

X₁(t) X2(t)

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

X1(t)

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

X2(t)

Simulation with P (Z ₁ = ± 1) = 1/2 and n = 2000 for both components.

−1.5 −1 −0.5 0 0.5

−1

−0.5 0 0.5 1

X₁(t) X2(t)

Simulations with Z ₁ ∼ N (0, 1) and n = 800 for both components.

−0.5 0 0.5 1 1.5

−1.5

−1

−0.5 0 0.5

X₁(t) X2(t)

−0.5 0 0.5 1 1.5

−1.5

−1

−0.5 0 0.5

X₁(t) X2(t)

−1 −0.5 0 0.5 1

−1.5

−1

−0.5 0 0.5

X₁(t) X2(t)

Literature

◮ Stochastic processes, measures on function spaces

V. I. Bogachev (1998), Gaussian Measures, Amer. Math. Soc., Providence.

I. Karatzas, S. E. Shreve (1999), Brownian Motion and Stochastic Calculus, Springer, New York.

M. A. Lifshits (1995), Gaussian Random Functions, Kluwer, Dordrecht.

L. Partsch (1984), Vorlesungen zum eindimensionalen Wiener Prozeß, Teubner, Leipzig.

N. N. Vakhania, V. I. Tarieladze, S. A. Chobanyan (1987), Probability

Distributions on Banach Spaces, Reidel, Dordrecht.

◮ Random numbers

J. E. Gentle (1998), Random Number Generation and Monte Carlo Methods, Springer-Verlag, New York.

P. L’Ecuyer (1998), Random number generation, in: Handbook on Simulation, J. Banks (ed.), Wiley, New York.

D. E. Knuth (1998), The Art of Computer Programming, Vol. 2, Seminumerical Algorithms, Addison-Wesley, New York.

G. Marsaglia (1995), The Marsaglia random number CDROM, including the DIEHARD battery of tests of randomness, Dep. of Statistics, Florida State Univ., Tallahassee.

H. Niederreiter (1992), Random Number Generation and Quasi-Monte Carlo

Methods, SIAM, Philadelphia.

Outline

I. Introductory Example: Reconstruction of a Brownian Motion Stochastic Processes

Brownian Motion

Reconstruction of a Brownian Motion

Reconstruction of a Brownian Motion X

Given n ∈ N , choose

knots 0 < t ₁ < · · · < t _n ≤ 1, functions a ₁ , . . . , a _n ∈ C ([0, 1]), such that X − X _n is ‘small’, where

X _n ( t ) =

n

X

i =1

X ( t _i ) · a _i ( t ).

Here mean-square L ₂ -distance. For every ω ∈ Ω k X ( · , ω ) − X _n ( · , ω) k ² =

Z ₁

0 | X (t , ω) − X _n (t , ω) | ² dt

^1/2 .

The (average) error of X _n is

e ( X _n ) = E( k X − X _n k ² 2 ) 1/2

.

n -th minimal error (minimization w.r.t. the a _i ’s and t _i ’s) e (n) = inf

X

e (X _n ).

Minimization of e ( X _n ) for fixed knots t _i : Recall e ² (X n ) =

Z

D

E(X (t ) − X _n (t )) ² dt . Put

Σ = ( K ( t _i , t _j )) _1≤ i , j ≤ n , b ( t ) = ( K ( t , t _i )) _1≤ i ≤ n , a ( t ) = ( a _i ( t )) _1≤ i ≤ n

to obtain

E(X (t ) − X _n (t )) ² = E(X ² (t )) − 2 E(X (t ) · X _n (t )) + E(X _n ² (t ))

= K ( t , t ) − 2

n

X

i =1

a _i ( t ) K ( t , t _i ) +

n

X

i ,j =1

a _i ( t ) a _j ( t ) K ( t _i , t _j )

= K ( t , t ) − 2 a ( t ) ^⊤ b ( t ) + a ( t ) ^⊤ Σ a ( t ).

Conclusion: With a ^∗ (t ) = Σ ⁻¹ b(t ) X _n ^∗ (t ) =

n

X

i =1

X (t _i ) · a ^∗ _i (t )

has minimal error among all X _n that use the knots t _i . Furthermore, e ² ( X _n ^∗ ) =

Z

D

K ( t , t ) − b ( t ) ^⊤ Σ ⁻¹ b ( t )

dt .

Valid for every second-order process X (if det Σ 6 = 0)!

Explicit formulas in the Brownian motion case: Here

Σ =











t ₁ t ₁ . . . t ₁ t ₁ t ₂ . . . t ₂ .. . .. . . .. ...

t ₁ t ₂ . . . t _n











= B C B ^⊤ ,

where, with t ₀ = 0,

B =













1 0 . . . 0

1 1 . .. ...

.. . . .. 0 1 . . . . 1













, C = diag(t _i − t _{i −1} ) _{1≤i ≤n} .

∗

Moreover,

e ² ( X _n ^∗ ) = 1 6

n

X

i =1

( t _i − t _{i −1} ) ² + 1

2 (1 − t _n ) ² .

Note: X _n ^∗ is ‘local’, which reflects the Markov property of X . Minimization of e ( X _n ^∗ ) with respect to the knots t _i :

Easy to see: e (X _n ^∗ ) is minimal iff

t _i ^∗ = 3 i

3 n + 1 .

Theorem (Suldin 1960) For piecewise linear interpolation X _n ^∗ with knots t _i ^∗

e ( X _n ^∗ ) = e ( n ) = 1

p 2(3n + 1) .

Used so far: m = 0 and K = min, but not: X Gaussian.

Notation: strong equivalence

a ( n ) ≈ b ( n ) if lim

n →∞ a ( n )/ b ( n ) = 1,

weak equivalence

Equidistant knots t _i = i / n yield e ( X _n ^∗ ) = 1

√ 6 · n ^−1/2 ≈ e ( n ).

Simulation: Clearly

Y _i = X (t _i ) − X (t _{i −1} ) ∼ N (0, t _i − t _{i −1} ), independent.

Use random numbers to simulate realizations of Y ₁ , . . . , Y _n .

Compare p. 3.

Reconstruction in L _p -norm: For 1 ≤ p ≤ ∞ and 1 ≤ q < ∞ , error of X _n

e _{p , q} (X n ) = "

E( k X − X _n k ^q p ) _1/ q

and n -th minimal error

e _p,q (n) = inf

X

e _p,q (X _n ).

So far p = q = 2. We have e _{p , q} ( n ) ≍

( n ^−1/2 if p < ∞ n ^−1/2 · √

ln n if p = ∞ .

Smoothness of a Brownian motion

◮ in mean-square sense:

q

E( X ( s ) − X ( t )) ² = | s − t | ^1/2 ,

◮ pathwise: almost surely (L´evy’s modulus of continuity) lim sup

h →0+

sup

|s −t |<h

| X ( s ) − X ( t ) |

p 2 h ln 1/ h = 1.

Decomposition of a Brownian motion

X − X _n ^∗ is zero mean Gaussian with covariance kernel

K ( s , t ) =



 

 



 

 



(min( s , t ) − t _{i − 1} ) ( t _i − max( s , t )) t _i − t _{i − 1}

if s , t ∈ [t _{i − 1} , t _i ] min(s , t ) − t _n if s , t ∈ [t _n , 1]

0 otherwise.

Moreover, X _n ^∗ and X − X _n ^∗ are independent.

A Gaussian process on [ u , v ] with covariance kernel

K ( s , t ) = (min( s , t ) − u ) ( v − max( s , t ))

Simulation of corresponding trajectories of

X , X _n ^∗ , X − X _n ^∗ .

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−0.5 0 0.5 1 1.5

t

W(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−0.5 0 0.5 1 1.5

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

W(t)−X(t)

Here n = 4 and

t ₁ = 1/4, t ₂ = 1/2, t ₃ = 3/4, t ₄ = 1.

Simulation of corresponding trajectories of

X , X _n ^∗ , X − X _n ^∗ .

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

W(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

W(t)−X(t)

Here n = 3 and

Simulation of corresponding trajectories of

X , X _n ^∗ , X − X _n ^∗ .

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

W(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

X(t)

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

−1

−0.8

−0.6

−0.4

−0.2 0 0.2 0.4 0.6 0.8 1

t

W(t)−X(t)

Here n = 4 and

t ₁ = 0.3, t ₂ = 0.4, t ₃ = 0.8, t ₄ = 1.

Literature

D. Lee (1986), Approximation of linear operators on a Wiener space, Rocky Mt. J.

Math. 16, 641–659.

K. Ritter (2000), Average-Case Analysis of Numerical Problems, Lect. Notes in Math.

1733, Springer, Berlin.

A. V. Suldin (1959, 1960), Wiener measure and its applications to approximation

methods I, II (in Russian), Isz. Vyssh. Ucheb. Zaved. Mat. 13, 145–158, 18, 165–179.