Separation of true from spurious breaks

On the multiple breakpoint problem and the number of significant breaks

in homogenisation of climate records

Separation of true from spurious breaks

Ralf Lindau & Victor Venema University of Bonn

Internal and External Variance

Consider the differences of one station compared to a neighbour or a

reference.

Breaks are defined by abrupt changes in the station-reference time series.

Internal variance within the subperiods External variance

between the means of different subperiods

Criterion:

Maximum external variance attained by a minimum number of breaks

Decomposition of Variance

n total number of years N subperiods

n_i years within a subperiod

The sum of external and internal variance is constant.

First Question

How do random data behave?

Needed as stop criterion for the number of significant breaks.

Random Time Series

with stddev = 1

Segment averages x_i scatter randomly mean : 0

stddev: 1/

Because any deviation from zero can be seen as inaccuracy due to the limited number of members.

ni

 ² -distribution

The external variance

is equal to the mean square sum

of a random standard normal distributed variable.

Weighted measure for the variability of the subperiods‘

means

From  ² to  distribution

n = 21 years k = 7 breaks

As the total variance is normalized to 1, a kind of normalized

chi²-distribution is expected:

This is the -distribution.

data

^



 



 



  

 

 

 

2 , 1 2 ) 1

(

2 1 1 1

2

k n

B k

v v v

p

k k n

The exceeding probability P gives the best (maximum) solution for v

Incomplete Beta Function













v

pdv v

P

0

1 ) ( 7 breaks in 21 years

Added variance per break

 5 ln 1 2

2 ln 1 1

1

*

*

*

*  



 



 

 





k k k

dk dv v k

 





 



 



 ¹ 

0

1 )

( ⁱ

l

l

l v m

l v v m

P

Incomplete -function:

2

3

 n m

2 i k

Transformation to dv/dk:

mean 90%

95%

The extisting algorithm Prodige

Original formulation of Caussinus and Mestre for the penalty term in Prodige

Translation into terms used by us.

Normalisation by k* = k / (n -1)

Derivation to get the minimum

In Prodige it is postulated that the relative gain of external variance is a constant for given n.

1  2 ln  min

ln v  k^* n 

  ⁰

ln 1 2

1

*  

  n

dk dv v

 ⁿ

dk dv

v 2ln

1 1

* 



  ^ln  ^min

1 1 2

ln 

 

 n

n v k

 

min )

1 ln(

2 )

(

) (

1 ln )

(

1

2 1

1

2

 

 























 







 n

n l k Y

Y Y Y n Y

C _n

i i

k

j j j k

Shorter length, less certainty

n = 21 years n = 101 years

Exceeding probability 1/128

1/64 1/32 1/16 1/8 1/4

Second Question

How do true breaks behave?

True Breaks

Identical Behaviour

True breaks behave identical to random data.

But the abscissa-scale is now:

k / nk instead of k / n.

Compared to random time series the external variance grows faster by the factor n / nk

data theory

n_k = 19 true breaks within n = 100 years time series

Assumed / True Break Number k / n_k

Break vs Scatter Regime

Simulated data with 19 breaks interfered by scatter

The internal variance decrease as a function of break number.

In the break regime the variance decrease faster by the factor:

15 breaks are detectable,

depending on signal to noise ratio.

Time series length Number of true breaks

Conclusions

• The analysis of random data shows that the external variance is -distributed, which leads to a new formulation for the

penalty term.

• True breaks are also -distributed. Their external variance increases faster by a factor of n/nk compared to random scatter.