Break signal

The break signal in climate records:

Random walk or random deviations?

Ralf Lindau

Dipdoc Seminar – 30. May 2016

Break signal

Climate records are affected by

breaks resulting from relocations or changes in the measuring

techniques.

For the detection, differences of neighboring stations are considered to reduce the dominating natural variance.

Homogenization algorithms identify breaks by searching for the

maximum external variance (explained by the jumps).

Benchmark datasets

Benchmarking data sets are used to assess the skill of homogenization algorithms.

These are artificial data sets with known breaks so that an evaluation of the algorithms is possible.

However, benchmark datasets should reflect as much as possible the statistical properties of real data .

An important question is how to model the breaks:

1. As free random walk (Brownian motion)

2. As random deviation from a fixed level (random noise)

Dipdoc Seminar – 30. May 2016

Conceptual model

Same signal, two approaches:

Which of the two DT is assumed to be an independent random variable?

The deviations or

the jumps?

Depending on our choice

different statistical properties of

break signal will result.

Dipdoc Seminar – 30. May 2016

Random deviations

Brownian motion

Different effects of identical s

Dipdoc Seminar – 30. May 2016

Difference:

The introduced break variance of random deviations is larger by a factor of 2 compared to Brownian motion.

Reason:

All jumps are created by the sum of two random numbers, while it is only one in case of Brownian motion.

Preliminary:

𝑽𝒂𝒓 _𝑩𝑴 = ^𝟏

𝟐 𝑽𝒂𝒓 _𝑹𝑫

Random deviations

Brownian motion

Linearly growing variance of BM

Dipdoc Seminar – 30. May 2016

The variance of a Brownian motion grows linearly in time.

At the end of a BM time series the variance is:

Var = k s² with k: number of breaks and s²: break variance

The average variance (over time) is Var_BM = k/2 s²

For RD the variance is (shown before):

Var_RD = 2 s²

k is in the order of 5, for difference time series twice: 10.

Thus k/4 is in the order of 2.5.

Brownian motion created by the same s is much easier to detect.

𝑽𝒂𝒓 _𝑩𝑴 = ^𝒌

𝟒 𝑽𝒂𝒓 _𝑹𝑫

Which type is more realistic?

Dipdoc Seminar – 30. May 2016

There are indications for both of the two break types:

For random deviations:

Relocations are bound to fixed position.

Stations have geographical names and their positions are not free to fluctuate away.

For random walk:

Changes in measuring techniques can be seen as elimination of error sources one after the other.

Ideal case: Today most errors are eliminated.

Then the break signal can be seen as Brownian motion backward in time.

Different “schools”

For a long time we were not aware that there are these two approaches.

Williams et al. (2012) modelled random walk.

Venema et al. (2012) modelled random deviations.

Only the standard deviations applied were communicated.

But these are not comparable for RD and BM.

Dipdoc Seminar – 30. May 2016

Platforms & Stairs

Venema et al. (2012) analyzed the statistics of the retrieved signal to decide whether breaks are BM or RD type.

Platforms Stairs

p (RD) = 0.67 p (actual) = 0.59 p (BM) = 0.50

But, the result was hardly significant due to the small number.

And (more important):

The result is dependent on the performance of the homogenization algorithm.

Dipdoc Seminar – 30. May 2016

T₃ T₁

T₂ T₃

T₁

T₂

Platforms are difficult to detect

Running a homogenization algorithm with artificial pure RD data results in 0.62 – 0.64 platform frequency ( < 0.67 ).

In the retrieved signal, the platforms are underestimated.

The detected frequency is not suited as independent indication parameter to distinguish RD from BM.

Therefore, it would be convenient to be independent from the retrieved break signal and instead able to derive break parameters directly from the data.

Dipdoc Seminar – 30. May 2016

Two superimposed signals

We assume that the climate time series consists of two superimposed signals:

Inhomogeneities and noise

𝑥 𝑖 = 𝜀_𝑏 𝑆 𝑖 + 𝜀_𝑛 𝑖 , 𝜀_𝑏 ~ 𝑁 0, 𝜎_𝑏² , 𝜀_𝑛 ~ 𝑁 0, 𝜎_𝑛²

Each yearly value can be thought as the sum of two random numbers, e_b and e_n, where e_b depends on segment number S, which is defined as the number of breaks lying temporally behind.

Dipdoc Seminar – 30. May 2016

Random deviation breaks

In case of random deviation breaks we calculate the

Lag-covariance C(L):

𝐶 𝐿 = 1

𝑛 − 𝐿 𝑥 𝑖 − 𝑥 𝑥 𝑖 + 𝐿 − 𝑥

𝑛−𝐿

𝑖=1

For external pairs E(C(L)) = 0 For internal pairs E(C(L)) = s_b²

𝐶 𝐿 = 𝑝_𝑖𝑛𝑡 ∙ 𝜎_𝑏²

Dipdoc Seminar – 30. May 2016

Probability of internal pairs

𝑝_{𝑦𝑒𝑎𝑟}(𝑙) = 𝑙 𝑘 + 1 𝑛

𝑛 − 1 − 𝑙 𝑘 − 1 𝑛 − 1

𝑘

𝑝_{𝑒𝑎𝑟𝑙𝑦} 𝑙 = 𝑙 − min(𝑙, 𝐿) 𝑙

𝑝_𝑖𝑛𝑡(𝐿) = 𝑝_{𝑦𝑒𝑎𝑟}(𝑙) ∙ 𝑝_{𝑒𝑎𝑟𝑙𝑦}(𝑙, 𝐿)

𝑛−𝑘

𝑙=1

Dipdoc Seminar – 30. May 2016

Probability of a specific year to belong to segment of length l:

Probability of a specific year to have sufficient spacing to the next break:

Probability of internal pairs is the sum over all length of the product.

The probability for internal pairs increase with segment length l and decrease with time lag L.

Probability of internal pairs

𝑝_𝑖𝑛𝑡 = 𝑙 𝑘 + 1 𝑛

𝑛−𝑘

𝑙=1

∙

𝑛 − 1 − 𝑙 𝑘 − 1 𝑛 − 1

𝑘

∙ 𝑙 − min 𝑙, 𝐿

𝑙

𝑝_𝑖𝑛𝑡 =

𝑛 − 1 − 𝐿 𝑛 − 1𝑘

𝑘

𝑝_𝑖𝑛𝑡 = 𝑒^{− 𝑛−𝑘𝑘𝐿}

Dipdoc Seminar – 30. May 2016

The long version of the product :

By some purely arithmetic transformations we get:

By some further approximations we get:

Lag covariance for RD

Dipdoc Seminar – 30. May 2016

The covariance is an

exponential function of the time lag.

C(L) = a exp (-bL)

break

a = s_b² strength s_b b = k/(n-k) number k

As byproduct we have a nice method to retrieve also

strength and number of breaks directly from the data.

Input:

s_b = 1.000 k = 5.000 Output:

s_b = 1.000 k = 4.984

Brownian motion type

Dipdoc Seminar – 30. May 2016

For Brownian motion type breaks the covariance depends only on the segment number of the earlier of the two years , because they have all random numbers e_b constituting the break signal at x(i) in common.

𝐶𝑜𝑣 𝑥(𝑖), 𝑥 𝑗 = 𝑆 𝑖 + 1 𝜎_𝑏² , 𝑖 < 𝑗

The segment number is a stochastic variable growing linearly in time:

𝑆 𝑖 = 𝑖 − 1

𝑛 − 1 𝑘 , 𝑖 ≤ 𝑛

Consequently, also the covariance grows linearly with time:

𝐶𝑜𝑣 𝑥 𝑖 , 𝑥(𝑗) = 1 + 𝑖 − 1

𝑛 − 1 𝑘 𝜎_𝑏² , 𝑖 < 𝑗 ≤ 𝑛

Time dependent Cov for BM

Dipdoc Seminar – 30. May 2016

The covariance is a linear function in time.

C(i) = a i + b

a = k/(n-1) s_b² b = ( 1 - k/(n-1)) s_b²

Input:

s_b = 1.000 k = 5.000 Output:

s_b = 1.005 k = 4.920

Mixed applications

Dipdoc Seminar – 30. May 2016

Very small break size.

Input:

s_b = 1.000 k = 5.000 Output:

s_b = 0.046 k = 14.95

Input:

s_b = 1.000 k = 5.000 Output:

s_b = 3.471 k = 0.729

Temporal covariance

Random deviations

Lag covariance Brownian motion

Very small break number.

Conclusion

Brownian motion and random deviation break types can be distinguished by calculating:

1. Lag covariance C(L)

2. Time dependent covariance C(i)

For Random deviations C(L) is decreasing with L.

For Brownian motion C(i) is increasing with j.

The two other combinations yield either small size or small number.

As byproduct we get an estimate for break size and number without running a full homogenization algorithm.

Dipdoc Seminar – 30. May 2016

Platforms & Stairs

Venema et al. (2012) analyzed the statistics of the retrieved signal to decide whether breaks are BM or RD type.

They distinguish platforms:

from stairs:

Dipdoc Seminar – 30. May 2016

T₃ T₁

T₂

T₃

T₁

T₂

Platform probability for RD

For RD break types T₁, T₂, T₃ are iid random variables (not the case for BM).

There are 6 possibilities of rank order, which all have the same probability:

Dipdoc Seminar – 30. May 2016

T₁ < T₂ < T₃ T₁ < T₃ < T₂ T₂ < T₁ < T₃ T₂ < T₃ < T₁ T₃ < T₁ < T₂ T₃ < T₂ < T₁

Upward and downward stairs have both the probability 1/6.

Every other combination is a

platform. (Either T₂ is the smallest or T₂ is the largest element of the

triple.)

Downward stair Upward stair

For RD break types the probability of platforms is 2/3.