4.1 An Example of Some Problems With Nonsampling Errors in Nontimber Surveys
George Gertner, Michael
Kohl
14.1.1 Introduction
Increasingly, we are required to monitor attributes in forest surveys that are so called "soft"
measurements. With the increased emphasis of nontimber attributes in forest surveys, these kinds of attributes are becoming more commonly assessed. These attributes tend to have a great deal of subjectivity in their assessment. If this subjectivity is overlooked in the analysis of the survey results, many complications can occur. In this paper, results are presented from a study of the subjective component in the evaluation of regional and national estimates of needle-leaf loss in Switzerland. The results are presented as an example of the problems that can occur when assessing such a variable.
In Europe, the assessment of percentage of foliage needle-leaf loss -, crown transparency is an equivalent expression of the attribute � was introduced in national forest health monitoring programs in the early I 980' s. It was assumed that crown transparency has a strong relationship to tree vitality.
Although this relationship has been questioned by some authors (for example, Brassel 1992), the assessment of crown transparency is still recommended by the United Nations Economic Commission for Europe, ECE. Today, in national reports within the European Community and the annually published ECE reports on forest condition, the main attribute reported on is needle-leaf loss. Many efforts have been undertaken to improve the quality of crown transparency assessments and reduce the subjectivity of observer error. However, considerable concern has been expressed regarding observer error (Neumann and Stowasser 1987; Innes 1988; Lick and Krapfenbauer 1986;
Kohl
1991 and 1993).Two error sources are associated with the assessment of crown conditions in sample surveys:
I . sampling error, and
2. observer error, which is defined as the difference between "true" and recorded crown transparency.
Check assessments taken during the Swiss National Forest Health Monitoring Program, suggest several factors effect observer error. Some factors include: tree species, social position, stage of development of the stand, and weather conditions during data assessment (Hagi 1989;
Kohl
1991).1 We are grateful to the former coordinators of the forest health monitoring program, Hans Rudolf Stierlin and Andreas Schwyzer, for their support and for making the data available for this study.
The findings from the check assessments have been used to improve the training of field crews. In Switzerland, an extensive workshop on the assessment of needle-leaf loss is conducted annually for the field crews. Additional workshops are held during the assessment period. Manuals have been written that define measurement techniques in detail (Stierlin and Walther 1988). A manual developed by Muller and Stierlin (1990) pictorially shows trees with different grades of needle-leaf loss for the major species and is used as a reference in the field assessment. Many European countries have developed similar reference manuals using the Swiss manual as a paragon. The purpose of holding workshops and writing the manuals is to make the evaluation of needle-leaf loss by the different field crews as consistent and uniform as possible. The ultimate goal is to remove all variability in needle-leaf loss due to observational error. However, in practice, this goal has not been achieved.
Needle-leaf loss is different from many traditional attributes that are measured in forest surveys. For many surveys, measurements are made of attributes that have exact values and can be directly measurable. These attributes, such as diameters and heights of trees, are usually measured with instruments. Although there might be instrument errors and possible ambiguity, there is likely to be a true value for these attributes. If resources are available, the true value can be determined. With needle
leaf loss, however, the measurement of this attribute is not directly measurable with instruments and requires the interpretation of the field crews. The accuracy of the assessment of needle-leaf loss depends on many different factors. In theory there is a true value for needle-leaf loss, but in practice, even if resources are available, it is not possible to determine the true value (Kohl 1991).
Field crew observational error, even with good and consistent training, can be a problem when determining the needle-leaf loss of a tree. 1l1is observational error, even when slight, can greatly reduce the precision of standard survey estimators. In this paper, two data sets from two separate studies are presented and evaluated. The first data set is limited; it is meant to present the general methodology. The second data set is based on a national survey for the entire country. A variety of species and attributes arc monitored in the national forest health surveys of Switzerland. Because of the importance of the species, discussion is limited here to the needle-leaf loss of Norway spruce (Picea abies (I.) karst.).
4.1.2 Data set one: test courses
A special investigation was planned and carried out during the field instructions for the Swiss National Health Monitoring Program in 199 1 . Crown transparency was recorded on special test courses where selected trees were assessed by nine field crews over several hours. Errors such as visibility of the crown, phenotype of the tree, tree species, social position and weather conditions were controlled or were considered constant for all trees in this study. A total of 167 Norway spruce trees were assessed on four test courses located in the cantons of Zug, Neuenburg, Ticino and Graubunden.
Nine two-person field crews assessed the courses independently. Communication between the different field crews was prohibited, but discussions within the crews was encouraged since this is common in practice. The errors made in needle-leaf loss were considered to be crew observation errors.
Among crews, observation errors were considered to be independent. It is asswned that all crews were equally skilled.
4.1.3 General Error Model
It will be assumed here that the test course trees that were measured were of a simple random sample of Norway Spruce. Let Yij= the observation of needle-leaf loss for the ilh tree by the jlh crew. For a sample size of n trees, the estimated mean needle-leaf loss for crew j would be Yi = ..:!. }: y1 . The "standard n 1=1 approach" for estimating variance due to sampling for a particular crew j would be,
(Eq. 1)
If a simple random sample of trees with replacement is assumed, the standard approach variance of the mean due to sampling for that crew would be
(Eq.2)
Table 1 shows the mean, variance due to sampling, and variance of the mean for each of the crews using the standard approach to obtain the estimates.
Table! . Test course estimates of needle-leaf loss by crew using standard approach.
Crew Number Mean Needle-Leaf Loss Variance due to Sampling Variance of Mean
j
-
� 2 (Standard)Yi O ui
V(yi )s1andard
1 17.485 143.034 .85649
2 15.210 172.998 1.03592
3 14.072 86.935 .52057
4 14.910 1 1 1.588 .66819
5 12.425 127.366 .76267
6 16.587 152.738 .91 460
7 16.497 143.227 .85765
8 17.635 145. 125 .86901
9 13.743 162.265 .97165
To better understand the consequences of observation error on the accuracy of estimates, it is useful to present an error model and the decomposition of mean square error based on the presented error model. The general error model was developed by Hansen, Hurwiz, and Bershad (1961). The error model they proposed is as follows.
Let Xi= true needle-leaf loss for the i.th tree sampled, and the conditional expectation of all possible
- 1 N
observations of a particular tree be Ei(Y1) = Yi· Let Y = N LY1 and where N is the number of trees in 1=1
the population.
For a particular crew j, the mean square error of the estimated mean would then be
(Eq.3)
where V(y1) is the variance of the mean based on crew j, and B is an overall bias across all crews, that is, the difference between the true mean and estimated mean across all possible crews.
In this paper, the emphasis will be on the properties of V(y1) . To estimate the overall bias, if it existed, would require that the true needle-leaf loss be known for a least a subset of trees. As already mentioned, it is not feasible to determine the true needle-leaf loss of a tree.
- - 1 n
It is possible to decompose the variance component V(y 1) . Let y = - L n 1=1 Y1 • Then
where
E{y
1-y}
2 is the observation error variance,E(y- v)
2 is the sampling variance, andE[2(y
1-yxy- Y)]
is the covariance between the observations and sampling deviations.Since no communication or consultation between crews were permitted about the needle-leaf loss of the individual trees, it is reasonable to assume that the covariance between the observations and sampling deviations,
E[2(y
1-yxy-Y)],
is equal to zero. This would not be a reasonable assumption if the different crews consulted with each other.The observation error variance,
E{y
1 -y)
2 , can be further partitioned. Let the deviation of individual tree observations be expressed as, d1 = y1 - Y1 • Then the observation variance can be written aswhere CJ � = E(df ) represents the variance due to observation and pa � = E(d,d1-, ) is the correlated component due to possible correlation of observations within a crew. The correlation, p , is the intrasample correlation, or for the needle-leaf loss study, the intracrew correlation.
The overall variance of the sample mean is then made up of the sum of the sampling variance, observation variance, and intracrew correlation variance components.
- CJ 2,
Let V(y1)sample = _n u be the sampling variance of the mean due to sampling of trees, where cr�.
represents the sampling variance of trees given error free observations,
V(y1)ooserv = � be the variance of the mean due to observation error, and n 2
- (n - 1 )pcr �
V(y1)corr observ = n be the variance of the mean due to intracrew correlated observation error.
The variance of the mean needle-leaf loss based on the observations of a single crew j can then be written as
(Eq.4)
When the standard approach is used to obtain estimates of the mean and variance, the usual assumption is made that there is independence of errors within a crew, that is p = 0 (Lessler and Kalsbeek 1992). If the intracrew correlation is relatively small and positive because of the subjective nature of the observations by crews, the consequences in terms of variance estimates can be serious. In the social sciences, there have been numerous studies of observational errors, most showing the very detrimental effects of such errors (Groves 1989). Some of the consequences due to positive correlation are as follows:
1. The variance due to observation error, V(y1)ooserv + V(y1)corr_observ , can be a significant component of the estimated variance of the mean.
2. If the simple random sampling estimators are employed when there is a positive correlation, the standard approach variance estimates will be biased. The variance estimate using the standard approach for estimation will usually be biased downward and will be less than V(Yi) . It can be shown that the variance estimate that will be obtained using the standard approach is
V(Y,)standard = V(Y,)sample + V(yf)observ
From this, the bias in variance can be shown to be approximately
3. Increasing sample size will not eliminate the problem with intracrew correlation. Even if the correlation is small, its significance can be large, particularly when n is large. As n goes to infinity, the variance of the mean will reach a minimum of po� , limit(V(y1)) = po� .
n➔-4.1.4 Estimation of Components and Analysis
To assess the consequences of observation error on the estimation of needle-leaf loss, a�- , a� , a� , and p had to be estimated using the test course data. It was possible to estimate the variance components and the intracrew correlations, since repeat observations were made of the trees. The approach used below to obtain the estimates of the variance components based on the repeat observations would remove the effects of the overall bias from the estimates if it existed (Cochran 1977).
Equation I was already used to calculate the variance due to sampling using the standard approach