Introduction to the BSRN Toolbox

(1)

QC offered by BSRN Toolbox H. Schmith¨usen

Outline Intro BSRN Toolbox Quality Checks Quality Codes Usage Demo

Quality Checks offered by the BSRN Toolbox

Holger Schmith¨usen

Alfred Wegener Institute

2012-08-01

(2)

Outline

1 Introduction to the BSRN Toolbox

2 Applied Quality Checks

3 Produced Quality Codes

4 Usage Demonstration of Quality Check Feature

(3)

(4)

Introduction to the BSRN Toolbox

Provided by WRMC to station scientists, data users and data curators

Open source (licensed under GPL) Main features:

Download manager for StationToArchive files Format converter (StationToArchive, Pangaea)

Basic file operations Quality Checks for BSRN data

(5)

Introduction to the BSRN Toolbox

Open source (licensed under GPL)

Main features:

(6)

Introduction to the BSRN Toolbox

(7)

Introduction to the BSRN Toolbox

Download manager for StationToArchive files

Format converter (StationToArchive, Pangaea)

(8)

Introduction to the BSRN Toolbox

(9)

Introduction to the BSRN Toolbox

Basic file operations

Quality Checks for BSRN data

(10)

Introduction to the BSRN Toolbox

(11)

Outline Intro BSRN Toolbox Quality Checks

Physically Possible Limits Extremely Rare Limits Comparisons Quality Codes Usage Demo

2 Applied Quality Checks Physically Possible Limits Extremely Rare Limits Comparisons

(12)

Applied Quality Checks

Current version of BSRN Toolbox provides one set of quality checks:

C. N. Long and E. G. Dutton:

BSRN Global Network recommended QC tests, V2.0

Physically possiblelimit tests Extremely rarelimit tests

Intercomparisons between different parameter

(13)

Physically Possible Limits

Theses parameter are checked for physically possiblevalues:

SW downwards SW diffuse SW direct normal SW upwards LW downwards LW upwards

(14)

Physically Possible Limits

SW downwards

SW diffuse SW direct normal SW upwards LW downwards LW upwards

UTC

[W m−2]

SWD

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 1.5Sa·cos(SZA)^1.2+ 100^W_m2

Min: −4_m^W2

(15)

Physically Possible Limits

SW downwards SW diffuse

SW direct normal SW upwards LW downwards LW upwards

UTC

[W m−2]

DIF

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 0.95Sa·cos(SZA)^1.2+ 50^W_m2

Min: −4_m^W2

(16)

Physically Possible Limits

SW downwards SW diffuse SW direct normal

SW upwards LW downwards LW upwards

UTC

[W m−2]

DIR

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: S_a Min: −4_m^W2

(17)

Physically Possible Limits

SW downwards SW diffuse SW direct normal SW upwards

LW downwards LW upwards

UTC

[W m−2]

SWU

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 1.2Sa·cos(SZA)^1.2+ 50_m^W2

Min: −4_m^W2

(18)

Physically Possible Limits

SW downwards SW diffuse SW direct normal SW upwards LW downwards

LW upwards

UTC

[W m−2]

LWU

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 700_m^W2

Min: 40^W_m2

(19)

Physically Possible Limits

UTC

[W m−2]

LWD

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 900_m^W2

Min: 40^W_m2

(20)

Extremely Rare Limits

Theses parameter are checked for extremely rarevalues:

(21)

Extremely Rare Limits

SW downwards

SW diffuse SW direct normal SW upwards LW downwards LW upwards

UTC

[W m−2]

SWD

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 1.2Sa·cos(SZA)^1.2+ 50_m^W2

Min: −2_m^W2

(22)

Extremely Rare Limits

SW downwards SW diffuse

SW direct normal SW upwards LW downwards LW upwards

UTC

[W m−2]

DIF

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 0.75Sa·cos(SZA)^1.2+ 30^W_m2

Min: −2_m^W2

(23)

Extremely Rare Limits

SW downwards SW diffuse SW direct normal

SW upwards LW downwards LW upwards

UTC

[W m−2]

DIR

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 0.95Sa·cos(SZA)^0.2+ 10^W_m2

Min: −2_m^W2

(24)

Extremely Rare Limits

SW downwards SW diffuse SW direct normal SW upwards

LW downwards LW upwards

UTC

[W m−2]

SWU

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: Sa·cos(SZA)^1.2+ 50^W_m2

Min: −2_m^W2

(25)

Extremely Rare Limits

SW downwards SW diffuse SW direct normal SW upwards LW downwards

LW upwards

UTC

[W m−2]

LWU

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 500_m^W2

Min: 60^W_m2

(26)

Extremely Rare Limits

UTC

[W m−2]

LWD

00 03 06 09 12 15 18 21 00

0 500 1000 1500

Max: 700_m^W2

Min: 60^W_m2

(27)

Comparisons

Comparisons are done for the following parameter:

SWDvs. DIF + DIR SWDvs. DIF

SWUvs. DIF + DIR LWDvs. T2

LWUvs. T2 LWDvs. LWU

(28)

Comparisons

SWDvs. DIF + DIR

SWDvs. DIF

UTC

[W m−2]

SWD vs. DIF+DIR

00 03 06 09 12 15 18 21 00

0 200 400 600 800 1000 1200

(29)

Comparisons

UTC

[W m−2]

SWD vs. DIF

00 03 06 09 12 15 18 21 00

0 200 400 600 800 1000 1200

(30)

Comparisons

SWUvs. DIF + DIR

LWDvs. T2 LWUvs. T2 LWDvs. LWU

UTC

[W m−2]

SWU vs. DIF+DIR

00 03 06 09 12 15 18 21 00

0 200 400 600 800 1000 1200

(31)

Comparisons

UTC

[W m−2]

LWD vs. T2

00 03 06 09 12 15 18 21 00

0 200 400 600 800 1000 1200

(32)

Comparisons

LWUvs. T2

LWDvs. LWU

UTC

[W m−2]

LWU vs. T2

00 03 06 09 12 15 18 21 00

0 200 400 600 800 1000 1200

(33)

Comparisons

UTC

[W m−2]

LWD vs. LWU

00 03 06 09 12 15 18 21 00

0 200 400 600 800 1000 1200

LWD

(34)

(35)

Produced Quality Codes

One QC per measured value

Designed as 6-bit array (decimal numbers 0−63) 2 bits per

physical possible limits extremely rare limits comparisons

extendable for additional future checks

(36)

Produced Quality Codes

above compa

red

measurement

belo wcompa

red

measurement

above extremely

rarelimit

belo wextrem

ely rarelimit

above physically

possible limit

belo wphysically

possible limit

bit position 5 4 3 2 1 0

decimal value 32 16 8 4 2 1

Examples

SWD= 9990_m^W₂ 0 0 1 0 1 0 = 10

LWD= 350_m^W2 1 0 0 0 0 0 = 32

LWU= 300^W

m² 0 1 0 0 0 0 = 16

(37)

Produced Quality Codes

above compa

red

measurement

belo wcompa

red

measurement

above extremely

rarelimit

belo wextrem

ely rarelimit

above physically

possible limit

belo wphysically

possible limit

Examples SWD= 9990_m^W₂

0 0 1 0 1 0 = 10

LWD= 350_m^W2 1 0 0 0 0 0 = 32

LWU= 300^W

m² 0 1 0 0 0 0 = 16

(38)

Produced Quality Codes

above compa

red

measurement

belo wcompa

red

measurement

above extremely

rarelimit

belo wextrem

ely rarelimit

above physically

possible limit

belo wphysically

possible limit

Examples

SWD= 9990_m^W₂ 0 0 1 0 1 0 = 10

LWD= 350_m^W2 1 0 0 0 0 0 = 32

LWU= 300^W

m² 0 1 0 0 0 0 = 16

(39)

Produced Quality Codes

above compa

red

measurement

belo wcompa

red

measurement

above extremely

rarelimit

belo wextrem

ely rarelimit

above physically

possible limit

belo wphysically

possible limit

Examples

SWD= 9990_m^W₂ 0 0 1 0 1 0 = 10

LWD= 350_m^W2

1 0 0 0 0 0 = 32

LWU= 300^W

m²

0 1 0 0 0 0 = 16

(40)

Produced Quality Codes

above compa

red

measurement

belo wcompa

red

measurement

above extremely

rarelimit

belo wextrem

ely rarelimit

above physically

possible limit

belo wphysically

possible limit

Examples

SWD= 9990_m^W₂ 0 0 1 0 1 0 = 10

LWD= 350_m^W2 1 0 0 0 0 0 = 32

LWU= 300^W

m² 0 1 0 0 0 0 = 16

(41)

Input Data Output Data Configuration Dialogue Visualisation

4 Usage Demonstration of Quality Check Feature Input Data

Output Data

Configuration Dialogue Visualisation

(42)

Input Data

Various input formats are supported:

Pangaea files as downloaded fromwww.pangaea.de

Output from Pangaea’sData Warehouse

TAB-delimited tables as produced by the format converter StationToArchive files need to be converted first

(43)

Input Data

Pangaea files as downloaded fromwww.pangaea.de Output from Pangaea’sData Warehouse

(44)

Input Data

TAB-delimited tables as produced by the format converter

StationToArchive files need to be converted first

(45)

Input Data

(46)

Output Data

Format of output files is Pangaea format

Available output: Quality codes

Auxiliary data (astronomical parameter, DIF+DIR) Original data set with flagged values removed

(47)

Output Data

Format of output files is Pangaea format Available output:

Quality codes

(48)

Output Data

Quality codes

(49)

Output Data

Quality codes

Auxiliary data (astronomical parameter, DIF+DIR)

Original data set with flagged values removed

(50)

Output Data

Quality codes

(51)

Output Data

Quality codes

(52)

Configuration Dialogue

(53)