Functional Equations & Neural Networks for Time Series Interpolation

Functional Equations & Neural Networks for Time Series Interpolation

Lars Kindermann, AWI Achim Lewandowski, OEFAI

Drop an object with different speeds and measure the speed at the ground

Question: What’s the speed

v

_m after half the way at x_m?

v₀ v₁

x₀

x₁

v₁ = f v ₀ v_m = ? x_m

v₀

v₀ v₁

free fall

friction Data

An old experiment

Free Fall:

Theory:

Model: with data fitted

With additional Friction:

Theory:

Model: Integrate numerically and fit and - already a non-trivial Problem!

t²

2



 x = g 

v₁ = f v ₀ = v₀² + 2g x g

t²

2



 x g k₁

t

– x k₂

t

x ²

– f

t

 x –  

= 

g k

Solving with traditional Physics

x₀

x₁

v₁ v₀

x₀

x₁

v_m = v_m x_m

v₀

v₁ = f v ₀

v₁ = v_m =   v₀



divide into two equal steps...

     v  = f v  

and solve this functional equation for  Theory: Assume translation invariance

A Data-based Aproach

A solution of this equation is a kind of square root of the function .

•

^{If :} is a function, we look for another function which composed with itself equals :

Because the self-composition of a function is also called

“iteration”, the square root of a function is usually called its iterative root.

is solved by the fractional iterates of a function :

   x  = f x 

 f

f x  IRⁿ  IRⁿ  x

f    x  = f x 

f f x   = f² x

ⁿ x = f^m x f

 x = f^{m n}  x

A Functional Equation

A solution of this equation is called a square root of .

•

^{If :} is a function, we look for another function which composed with itself equals :

Because the self-composition of a function is also called “iteration”, the square root of a function is usually called its iterative root.

is solved by the fractional iterates of a function :

   x  = f x 

 f

f x  IRⁿ  IRⁿ  x

f    x  = f x 

f f x   = f² x

ⁿ x = f^m x f

 x = f^{m n}  x

= ^f _

 x

x

y y

= ^f

 x

x

y

    y

f

3 5---

A Functional Equation

The exponential notation of the iteration of functions can be extended beyond integer exponents:

•

^means

•

for positive integers are the well known iterations of

•

denotes the identity function,

•

is the inverse funktion of

•

is the -th iteration of the inverse of

•

is the -th iterative root of

•

is the -th iteration of the -th iterative root or fractional iterate of The family forms the continuous iteration group of .

Within this the translation equation is satisfied.

fⁿ x

f ¹ f

f ⁿ n f

f⁰ f⁰ x = x

f ^–1 f

f ^–n n f

f ^{1 n} n f

f^{m n} m n f

f^t x f

f ^{a b+}  x = f ^af ^b x 

Generalized Iteration

Map this to a Network

 

x

 x

f x 

f

share weights

f^{1 n} x

 = f^{m n}

f x 

  

1 m n

            

x f x 

loop n times

•

Weight Copy: Train only the last layer and copy the weights continously backwards

•

Weight Sharing: Initialize corresponding weights with equal values and sum up all delivered by the network learning rule

•

Weight Coupling: Start with different values and let the corresponding weights of the iteration layers approach each other by a term like

•

Regularization: Add a penalty term to the error function which assigns an error to the weight-differences to regularize the network. This allows to uti- lize second order gradient methods like quasi Newton for faster training.

•

Exact Gradient: Compute the exact gradients for an iterated Network w_i



w_i

 = w_j – w_i

Training Methods

0 1 2 3 4 5 6 7 8 9 10 0

1 2 3 4 5 6 7 8 9 10

Start Velocity [m/s]

End Velocity [m/s]

measurement for v1 (training data) physics for vm (prediction task) fractional iterates (network results)

v₁ = f v ₀

f ⁰ v₀ = v₀ v_m = f ^{1 2}  v₀ f ^{1 4}

f^{3 4}

0 1 2 3 4 5 6 7 8 9 10

0 1 2 3 4 5 6 7 8 9 10

Start Velocity [m/s]

End Velocity [m/s]

measurement for v1 (training data) physics for vm (prediction task) fractional iterates (network results)

v₁ = f v ₀

v_m = f ^{1 2}  v₀ f ⁰ v₀ = v₀

f^{3 4} f ^{1 4}

no friction with friction

The Network results are conform to the laws of physics up to a mean error of 10^-6

Results for „The Fall“

0

0.2

0.4

0.6

0.8

1

1.2

0 1 2 3 4 5 6 7 8 9 10

0 2 4 6 8 10 12

h [m]

v0 [m/s]

v [m/s]

Training Data

trajectories iterative roots

v

₀

v(h)

v=f(v

₀

,h)

h

The Embedding Problem

One of the most important functional equations:

The Eigenvalue problem of functional calculus.

Transform to:

  f x    = c    x

f x  = ^–1c x  invert

 ^–1

x

 x

f x  c

train

            

f

The Schröder Equation

Commuting Functions

f x 

f x  x

x



 f

f weight

sharing

outputs targets

 x

  f x    = f     x 

-

The steel bands are processed by N identical stands in a row

-

^, are known and can be measured

-

                       x₂

x₁ x_out

    

     f x _in,p₁ f x ₁,p₂

F x _in,p₁p_N

     f x _{N 1–} ,p_N

x_in

Measuring

p₁ p₂ p_N

instrument

=set of parameters like force, heat, strip thikness and width...

pi

x_in p_i x_out

x_out = F x _in p₁...p_N = f ...f f x   _in p₁ p₂... p, _N

Steel Mill Model

Steel Mill Network

For a given autoregressive Box-Jenkins AR(n) timeseries , we

define the function : which maps the vector of the last n samples

one step into the future as

and can simply write now.

The discrete time evolution of the the system can be calculated using the

matrix powers of F: .

x_t a_kx_{t k–}

k = 1 n



=

F Rⁿ  Rⁿ

x_{t 1–} = x_{t 1–}   x_{t n–} x_t = x_t x_{t 1–}   x_{t– n 1– }

F

a₁ a₂  a_n 1 0 0 0 0 1 0 0 0 0 1 0

= x_t = F x _{t 1–}

x_{t n+} = Fⁿ  x_{t 1–}

Timeseries Interpolation

This autoregressive system is called linear embeddable if the matrix power exists also for all real . This is the case if can be decomposed into

with being a diagonal matrix consisting of the eigenvalues of and being an invertible square matrix which columns are the eigenvec- tors of . Additionally all must be non-negative to have a linear and real embedding, otherwise we will get a complex embedding.

Then we can obtain with

Now we have a continuous function and the interpolation of the original time series consists of the first element of .

F^t

t  R⁺ F

F = S A S  ^–1 A _i

F S

F _i

F^t = S A ^t  S^–1 A^t

₁^t 0 0

0  0

0 0 _n^t

=

x t  = F^t  x₀

x t  x

Using Generalized Matrix Powers

The Fibonacci series , , is generated by and . By eigenvalue decomposition of we get

That is Binet’s formula in the first component

x₀ = 0 x₁ = 1 x_t = x_{t 1–} + x_{t 2–}

F 1 1

= 1 0 x₁ = 1 0  F

x_{t 1+} = F^tx₁ = SA^tS^–1x₁

1+ 5

--- 12 – 5 ---2

1 1

1+ 5

---2

 

 t

0 0 ¹---^–₂ ⁵t

1

---5 1

2--- 1

2 5

--- –

1

5

--- –

1

2--- 1

2 5

---

+

1

= 0

x_t 1

--- 15 + 5 ---2

 

 ^t 1 – 5

---2

 

 ^t

–

=

Example: A continuous Fibonacci Function

A time series of yearly snapshots from a discrete non linear Lotka-Volterra type predator - prey system (x = hare, y = lynx) is used as training data:

and

From these samples we calculate the monthly population by use of a neural network based method to compute iterative roots and fractional iterates.

The given method provides a natural way to estimate not only the values over a year, but also to extrapolate arbitrarily smooth into the future.

x_{t 1+} = 1 a b y+ – _t x_t y_{t 1+} = 1 c– + d x_t y_t

Nonlinear Example

0 2 4 6 8 10 12

0 1 2 3 4 5 6 7

Prey

Predator