Detection of Breaks in Random Data, in Data Containing True Breaks, and in Real Data

Detection of inhomogeneities in Daily climate records to Study Trends in Extreme Weather

Detection of Breaks in Random Data,

in Data Containing True Breaks, and in Real Data

Ralf Lindau

Internal and External Variance

Consider the differences of one station compared to a neighbour or a

reference.

Breaks are defined by abrupt changes in the station-reference time series.

Internal variance within the subperiods External variance

between the means of different subperiods

Criterion:

Maximum external variance attained by

a minimum number of breaks

Decomposition of Variance

n total number of years N subperiods

n

years within a subperiod

The sum of external and

internal variance is constant.

Three Questions

How do random data behave?

Needed as a stop criterion for the number of significant breaks.

How do real breaks behave theoretically?

How do real data behave?

Segment averages

with stddev = 1

Segment averages x

scatter randomly mean : 0

stddev: 1/

Because any deviation from zero can be seen as inaccuracy due to the limited number of members.

n

 ² -distribution

The external variance

is equal to the mean square sum

of a random standard normal distributed variable.

Weighted measure for the variability of the subperiods‘

means

From  ² to  distribution

n = 21 years k = 7 breaks

data

 ^

X ~  ² (a) and Y ~  ² (b)  X / (X+Y) ~  (a/2, b/2)

If we normalize a chi

-distributed variable by the sum of itself and another chi

-distributed variable, the result will be -distributed.



) (

) ( ) ) (

,

( a b

b b a

a

B  



 

 

 

 



  

  ^



 

2 , 1 2 ) 1

(

2 1 1 1

2

k n

B k

v v v

p

k k n

with

Incomplete Beta Function

 

 

 



  

  ^



 

2 , 1 2 ) 1

(

2 1 1 1

2

k n

B k

v v v

p

k k n

External variance v is -distributed

and depends on n (years) and k (breaks):

2

i  k  ^  



 

 

 



 ¹ 

0

1 )

( ⁱ

l

l

l v m

l v v m

P

Solvable for even k and odd n:

2

 3

 n m

The exceeding probability P gives the best (maximum) solution for v

Incomplete Beta Function





 





v

pdv v

P

0

1 ) (

We are interested in the best solution, with the highest external variance.

We need the exceeding probability for high var

P(v) for different k

Can we give a formula for in order to derive v(k)?

2

20 breaks

dk dv

Increasing the break number from k to k+1 has two consequences:

1. The probability function changes.

2. The number combinations

increase.

dv/dk sketch

P(v) is a complicated function and hard to invert into v(P).

Thus, dv is concluded from dP / slope.

And the solution is:

k b re ak s

k+ 1 b re ak s

 

 ^



 

 

 



 ¹ 

0

1 )

( ⁱ

l

l

l v m

l v v m

P

   

  ^ _ ^

 

  

 









 

 

 



  





 

v k

v k n

k k c n

k n

v dk

dv

1 ln 1

2 1 ln 1

1

1

2

Solution

  5 ln 1 2

2 ln 1 1

1

*

*

*

*

 



 



 

 





k k k

dk dv v k

 

*

*

*

1 5 ln 1 2

2 ln 1 1

1 dk

k k

dv k

v  





 





 

 

 



  



 ^{1 *}  ^{2 ln( 5 ) 2 1 1} _* ^{* 2 1}

1

k

k k k

v



  

 



  





Constance of Solution

10 1 y e ar s 21 y ea rs

The solution for the exponent 

is constant for different length of

time series (21 and 101 years).

The extisting algorithm Prodige

Original formulation of Caussinus and Mestre for the penalty term in Prodige

Translation into terms used by us.

Normalisation by k* = k / (n -1)

Derivation to get the minimum

In Prodige it is postulated that the relative gain of external variance is a constant for given n.

 1  2 ln   min

ln  v  k

n 

  ⁰

ln 1 2

1

*

 

  n

dk dv v

  ⁿ

dk dv

v 2 ln

1 1

*





  ^ln   ^min

1 1 2

ln 

 

 n

n v k

 

min )

1 ln(

2 )

(

) (

1 ln )

(

1

2 1

1

2

 

 

 



 



 



 









 









n

n l k Y

Y Y Y n Y

C

i i

k

j j j k

Our Results vs Prodige

We know the function for the relative gain of external variance.

Its uncertainty as given by isolines of exceeding probabilities for 2 ^-i are characterised by constant distances.

Prodige propose a constant of 2 ln(n) ≈ 9

Exceeding probability 1/128

1/64 1/32 1/16 1/8 1/4

Wrong Direction

n = 101 years n = 21 years

True Breaks

Only true for constant lengths

True breaks with fixed distances behave identical to random data.

For realistic random lengths the exponent is slightly increased.

Sub-periods with random lengths Sub-periods with

constant lengths

data theory

theory data

Distribution of Lengths

The distribution of the sub- periods’ lengths as obtained by randomly inserted breaks is known.

If necessary, it could be

taken into account.

Break vs Scatter Regime

The two governing parameters are:

1) The relative amount of break variance compared to the scatter variance

2) The quotient

The latter defines how much faster the internal variance decreases in the “true break regime”

compared to the “scatter regime”

If the relative scatter is low (10%) the transition between the regimes is clearly visible at 15 from 19 breaks.

Time series length

Number of true breaks

Real Data

1050 Climate Stations exist in Germany.

For each station the next eastward (to avoid identical pairs) neighbour between 10 km and 30 km is searched.

443 stations pairs remain.

All Stations Neighbouring pairs

Data Focus

This project deals with daily climate data.

Findings about their extremes are in the focus.

At least statements about the

•distribution (moments)

• percentiles

• indices (number of wet days per month)

should be possible.

Parameters

Interesting for break detection:

Problem parameters PP

Expected physical problems

Temperature at high sun shine duration Temperature at high pressure

Temperature at high diurnal cycle Temperature during snow cover Temperature depending on general

weather situation Temperature during rain

Rain at high wind speed

Expected technical problems

Frequency of rainy days below 1 mm Tenth of precipitation report

Difference between T

and (T

-T

)

Per se interesting parameters P

Monthly means

Temperature Precipitation, etc.

Breaks are more sensitive to problem parameters. Breaks in PP may help to find breaks in P

Distribution and extremes

Standard deviation Skewness

Kurtosis Maximum Minimum 90 percentile

project focus

(more sensitive?)

Two Parameter Pairs

1a. Monthly mean temperature 1b. Monthly maximum temperature 2a. Monthly precipitation sum

2b. Frequency of rainy days below 1 mm

Can the sensitive parameter help to find breaks in the mean?

(Project focus)

(Problem parameter)

“Drizzle days” are often excluded from rainy days to calculate the interesting indices:

•Monthly Rain Frequency

•Consecutive Dry Days

“Drizzle frequency” is not only a

technical problem parameter, but

also a per se interesting one.

Monthly Mean Temperature

Temperature difference between

Ellwangen-Rindelbach and

Crailsheim-Alexandersreut shows 1 strong and

3 further significant breaks.

The statistical signature confirms it:

The first break contains much variance.

2, 3 and 4 are only

slightly larger than the

Mestre penalty.

Break Statistics

Individual pair All pairs

r = 0.937

Monthly Maximum

For the monthly temperature maximum, only the largest breaks are detectable, probably due to the reduced correlation.

r = 0.865

Additional Breaks?

In maximum temperature there are less breaks. Are they nevertheless new

compared to those in mean temperature?

Enhance the penalty from about 12 (i.e. 2 ln(n)) to 60.)

With n = 600, it means that 10% of the remaining internal variance has to be explained by each additional break.

Otherwise the search is stopped.

For such increased requirements 297

breaks are found in the mean and 67 in the maximum.

Nearly all breaks in t

exist also in t

.

The “stddev” of temporal distance is 1.75

years.

Answer: No

Nearly no new break is found by the sensitive parameter Monthly Maximum Temperature.

The lower correlation (0.865 vs. 0.937  doubled rms) hamper obviously the break finding capability of the sensitive parameter.

However, the high correlation of break positions may the opposite direction become possible: To find break

positions in the maximum temperature by considering the

mean temperature.

“Drizzle Days”

Monthly frequency of rainy days below 1mm.

This parameter is highly inhomogeneous.

Even for individual stations the break is

evident.

Drizzle vs. Mean Precip.

In the drizzle parameter more significant breaks are found (index 43.3 compared to 28.8), although the correlation is low, (0.339 compared to 0.855).

Are the break positions again

correlated?

Correlation of break positions

Many new breaks are found. Only 12 breaks of the drizzle parameter are found at all somewhere the corresponding time series of mean precipitation, but mostly far away.

In 93 time series pairs one or more breaks are found for drizzle, but even not a single in mean precipitation.

Are these new breaks also included, but hidden in mean precipitation?

 remember

Forced Breaks (1)

Forced Breaks (2)

Also in average, the external variance decreases only by about 1%, if “drizzle breaks” are inserted into the time series of mean precipitation.

1% is the mean decrease of a random n=100 time series and it is beta-

distributed.

However, here n is equal to 600. Is the

result then a bit better than random?

Simulated Data

1. Blind try of 3 breaks in a 21 years random time series

2. Blind try of 3 breaks in a 21 years constant time series with 6 true breaks.

3. Blind try 3 breaks in a 21 years time series with 6 true breaks plus random scatter.

1. Purely random 2. Pure true breaks 3. Realistic mix

Realistic Mixed Data

Real data is expected to be similar to a realistic mix, rather than to random scatter.

As it then includes also real breaks, the Null Hypothesis is not random scatter, but a realistic mix.

Here the blindly found external variance is again -distributed, but generally larger. How

much is difficult to quantify in

advance . It depends on the

signal to noise ratio.

Conclusions

• The analysis of random data shows that the external variance is -distributed, which leads to a new formulation for the penalty term.

• True breaks are also -distributed. Their external variance increases faster by a factor of n/n _k compared to random scatter.

• Are sensitive parameters helpful to find additional breaks?

Monthly maximum temperature:

Due to the reduced spatial correlation T max “finds” less breaks.

Those identified are even better visible in T mean . Drizzle parameter:

Highly inhomogeneous  Many breaks found.

But they do not coincide with breaks in mean precipitation.

• Vice versa we expect that T mean breaks are helpful to find breaks in T max . But

the prove of significance will be difficult.