Chapter 8: Investigating Long Term Teacher Learning
8.2 The studies .1 Introduction
8.2.4 Study 2: Sorting task
8.2.4.1 Introduction
This study investigates whether teachers have acquired a network of practice-specific implicit knowledge using a sorting task. This type of task has been traditionally used to look at knowledge organization in a variety of disciplines (Chi, Feltovich & Glaser, 1981; Freyhof, Gruber & Ziegler, 1992; Gess-Newsome & Lederman, 1993; Gruber &
Ziegler, 1990, 1993; Jones & Vesilind, 1996; Leinhardt & Smith, 1985; Llinares, 2000;
Nathan & Koedinger, 2000; Stein, Baxter & Leinhardt, 1990; Tamir, 1992). In the sort task participants are generally asked to sort the stimulus material into any groups that make sense to them (for example by circling the groups given on one piece of paper or sorting stimulus materials of cards into piles) and to give each group a label. While the groups that the participants make are not thought to directly represent how they group their knowledge, it is thought that people use their knowledge structures to solve such tasks and that the results reflect their organization of tacit knowledge. For example, the classic study by Chi and her colleagues (Chi, Feltovich & Glaser, 1981) found that novices sorted the sample physics questions into groups based on superficial characteristics (i.e. “problems with a slope“) while experts used structurally deeper categories to sort the stimulus material (i.e. “conservation of energy“).
8.2.4.2 Procedures
In this study the participants were presented with 30 English sentences written by hand on index cards. The sentences had been selected from essays by pupils in a German gymnasium (high school). The selection criteria were (1) that the sentences should be understandable alone without any other contextual information and (2) that the sentences represent a wide range of correctness, i.e., correct sentences (The government should raise taxes or pass stricter laws) as well as a range of problems with usage of English grammar (I didn’t felt me well yesterday), lexical knowledge (This is one reason because I do not become a politician), and pragmatics (Hallo. I will interview you to see what you do to help protect the environment). As this was an exploratory study, the widest variety of sentences possible was used in order to access as wide a range of knowledge structures as possible. On the back of each card was a letter code to make the recording of the data easier.
The participants were instructed to sort the cards into groups that made sense to them in terms of analyzing pupil’s language and then to give each group a label or definition.
The participants were then asked if there was any way to combine the groups they had already made to form bigger metagroups. When this was finished they were instructed to try and divide their groups into still smaller groups. This continued until the groups contained fewer than 3 cards or the participants could not divide a group further. The group labels and the cards in each group were recorded at each stage in the process.
8.2.4.3 Data analysis
The first step was to represent the categories and their connections in graphical form by making hierarchy maps which represent the groups and subgroups that the participants formed. Below is an example of such a hierarchy map.
The primary analysis is qualitative as is the case with other studies using sorting tasks (Chi, Feltovich & Glaser, 1981; Freyhof, Gruber & Ziegler, 1992; Gess-Newsome &
Lederman, 1993; Gruber & Ziegler, 1990; Leinhardt & Smith, 1985; Stein, Baxter &
Leinhardt, 1990). The hierarchy maps were analyzed for coherence, for links between the groups (as was done in Chi, Feltovich & Glaser, 1981; Freyhof, Gruber & Ziegler, 1992;
Gruber & Ziegler, 1990; Leinhardt & Smith, 1985) and for the ways in which categories were specific to the activity of teaching (as in Gess-Newsome & Lederman, 1993; Stein, Baxter & Leinhardt, 1990).
Figure 8.1: Katja’s Hierarchy Map
A number of quantitative measures were also used to support the qualitative analysis.
The number of categories (the groups made with the cards) and nodes (where one category is subdivided into others) were measured in order to measure the complexity and structure of the hierarchy maps. (See figure 8.1 for an example.) A content analysis labeled categories as being “linguist categories” (ones that focused exclusively on language analysis) and “teacher categories” (ones that focused on teaching or material for teaching). The remaining categories were classified as “Other”. This was done by the author and another experienced teacher. The interrater reliability rate for the Linguist categories was 89% and for the Teacher categories it was 92%. It was also investigated which stimulus sentences served as prototypical models for each of these categories. A stimulus sentence was counted as prototypical for one of the participant groups if 25% or more of a participant group used it in a particular category.
8.2.4.4 Results and discussion
The qualitative analysis revealed no significant differences between the hierarchy maps the teachers made and those that the other participants made. The maps mainly consisted of different parts of speech, but even here these did not seem to be organized in deep, teaching-specific ways. There was little teacher specific organization or reference to actions teachers could take. Furthermore, there often seemed to be a lack of cohesion in teachers’ maps. An example can be seen in Katja’s hierarchy map above (fig. 8.1).
“Wrong Vocabulary” is one of the categories under “Mistakes”, but “Vocabulary” is also a category under “Bad Mistakes”. Overall this data leaves the impression of a knowledge base focused only on identifying and describing mistakes, but no more.
Vocab
Active/
Passive Wrong Usage of
Verb Forms Verb Form Bad Mistakes
Have to learn set phrases
Predicate Problems
Prep. Much/
Many
Personal pronoun Single
Word Problems
Have to look up Minor
Mistakes
Object in wrong
place
Subject
&
Predicate Double Subject Word
Order
Similar to another
word
Thinking the German
way
Not same meaning as other word Mixing
up known words
Leaving out letter
Incorrect Spelling Wrong
Vocab Mistakes
General Sentences
Society
&
People No Problems Language Samples
8.2.4.4.1 Size and complexity
The teachers and linguists produced about the same number of categories (see table 8.1), with the teachers an average of 21.58, the linguists producing an average of 20.84 and the control group only averaged 15.27 categories per person. These differences between the teacher and the linguists, on the one hand, and the control group, on the other, were statistically significant (Mann-Whitney Test, p < .05). There was little variation between the participant groups in terms of categories per node, meaning that the linguists and teachers made more complex maps out of the content, rather than simply having a greater number of categories within a small conceptual map of the content.
Table 8.1 Categories and Nodes
Categories Nodes
Mean SD Mean SD
Teachers 21.58 6.25 7.37 2.45
Linguists 20.84 6.20 6.79 2.88
Control 15.27 5.50 5.44 3.18
The extent to which the categorizations were a product of the stimulus materials was also investigated. It has been shown that when experts are confronted with data that are unclear or fuzzy, they use their knowledge organization to impose an order the data (Lesgold, Rubinson, Feltovich, Glaser, Klopfer, & Wang, 1988). In other words, does the data reflect the schemata of the participants or was the data more defined by the stimulus materials than the participants own knowledge organization? As can be seen in table 8.2, the stimulus did not significantly restrict the participants’ categories. All of the stimulus sentences were placed in a wide variety of categories by all of the participant groups. In fact, the majority of all stimulus material was put into a category by only one or two people from one participant group. For example, the stimulus sentence “H“ was put into the category “Word Order“ by 3 teachers but not by any linguists or members of the control group.
Table 8.2 Frequency of stimulus sentence representing a category for a participant group
Number of people in a participant group to classify a particular stimulus sentence in a particular category
1 2 3 4 5 6 7 8 9 10+
Number of occurrences 198 59 63 42 18 21 20 14 12 20
8.2.4.4.2 Categories
As can be seen in table 8.3, the teachers produced more “Teacher” categories than the other groups, a difference which is highly statistically significant (Mann-Whitney Test, p
< .005). However, the teachers’ percentage of “Teacher” categories compared with their total number of categories was almost the same as the control group (32.9% vs. 27.6%).
In other words, the teachers formed more “Teacher” categories than the control group because they formed more categories in general, not because they were more likely to make a “Teacher” category than other participant groups. The other categories are similar in this respect. Teachers’ percentage of “Linguist” categories was nearly halfway between the other two participant groups (i.e., teachers formed a greater percentage of
“Linguist” categories than the control group but fewer than the linguists). The control group had the highest percentage of “Other” categories, but the teachers and the linguists
formed such groups with almost equal frequency. This data supports the results of qualitative analysis in that it shows very little difference between the categories teachers made and those of the other groups. While there were differences with one group or the other, teachers were always similar to one group or they were between the two other groups. In no way did teachers form groups that were different from both the other groups at the same time.
Table 8.3 Percentage Teacher and Linguist categories
Teacher Cat.
Mean
Teacher Cat.
as % of total
Linguist Cat.
Mean
Linguist Cat.
as % of total
Other Cat.
mean
Other Cat.
Teachers 7.11 32.9% 11.42 52.9% 3.05 14.2%
Linguists 1.05 5.0% 16.95 81.3% 2.84 13.7%
Control 4.22 27.6% 5.83 38.3% 5.22 34.1%
There were five main teaching categories in the data the participants produced: (1) Information about Students (e.g., “Advanced Level”, “Shows Pupils Can Go Into Detail”, “More/Less important”, “Typical Mistake”, etc.), (2) Opportunities For And Ways of Teaching (e.g., “Should Be Taught More Carefully”, “I Never Teach This”,
“Opportunity To Talk About Relative Pronouns”), (3) Materials for Teaching (e.g.,
“Language Needed For Themes”, “Example Of Comparing Things”, “Ways to Express Facts”), (4) Marking Students’ Work (e.g., “Would Mark Wrong”, “Would Mark As A Style Problem”), and (5) Need Secondary Material (e.g., “Have To Look Up”, “Would Look Up”).
Table 8.4 Kinds of Teaching categories Information
about Students
Opportunities For & Ways Of Teaching
Material For Teaching
Marking Need Secondary Material Teachers 11 (58%) 7 (37%) 11 (58%) 2 (11%) 2 (11%)
Linguists 8 (42%) 3 (16%) 6 (32%) 0 2 (11%)
Control 6 (33%) 3 (17%) 8 (44%) 0 0
There was only one “Teacher” category which the teachers used exclusively, “Marking“, and this was only used by 11% of the teachers. Even including this example, none of the differences between number of teachers using any of the “Teacher” categories and the number of linguists was statistically significant (Mann-Whitney Test, p < .05).
8.2.4.4.3 Prototypical stimulus sentences
The teachers and the other participant groups show no significant differences in terms of the general categories they form. Nevertheless, it is possible that teachers have a much more specific idea of what their categories mean than the linguists or the control group.
Therefore, the prototypical sentences for categories were compared between the participant groups. A stimulus sentence was counted as prototypical for one of the participant groups if 25% or more of a participant group used it in a particular category.
There was said to be agreement that a sentence was prototypical for two participant groups if both groups had at least 25% of its members classify a sentence into that category but the difference between the groups was less than 25 percentage points. For
example, 26% put stimulus sentence “B” into the “Correct” category, but 53% of linguists classified “B” as correct – a difference of 27 percentage points – so “B“ was classified as being prototypical of the category “Correct” for the linguists only. Stimulus sentence “P”, however, was placed in the “Correct” category by 53% of the teachers, 58% of the linguists, and 50% of the control group. Therefore, it was classified as being prototypical of “Correct” for all three of the participant groups.
Table 8.5 Prototypical stimulus sentences
Teachers & Linguists Teachers Only Linguists Only
Categories Sentences Example Sentences Example Sentences Example Correct P, R, S, Z The
government should raise taxes or pass stricter laws
A, B, G, O, Ü
This family is no real family
Grammar F, H, I, K, O, Ö, T, V, W, X, Y ß
I was caught 3 fishs
U The American pupils they stay a long time in school Word Order C, I, In every
country are politics a dirty business
A, U My mum is a secretary, what my dad is I can’t say Form F I didn’t felt
me well yesterday
Ö We haven’t
saw Mickey or Minnie Vocabulary N, Q Hallo. Can I
have some questions to you?
G, M, T, V Hallo. I will interview you to see what you do to protect the environmen t
Relative Clauses
T My father and I saw a cake who was as expensive as 1000DM
Table 8.5 shows that within the more general categories like “Correct”, “Grammar”,
“Vocabulary”, and “Form” there is broad agreement between the teachers and the linguists on what constitutes a prototypical example of that category. This was also the case for more the specific categories “Word Order” and “Relative Clauses”, which were often used by both teachers and linguists. In addition, the teachers and linguists had other prototype sentences in these categories that they did not share with the other group; the teachers in “Grammar”, “Word Order” and “Vocabulary”, and the linguists in the
categories “Correct” and “Form”. There were a few categories where only the teachers or only the linguists had prototypical sentences; however, this was mainly the case for the linguists. The teachers had only two exclusive categories “Tense” and “Passive” and each only contained one prototypical sentence. The linguists, on the other hand, had six exclusive categories with 15 prototypical sentences, and average of 2.5 per category. In summary, the teachers’ prototypical categories were not significantly different from those of the linguists. This data further supports the notion that the teachers in this study have not acquired a rich, practice-specific network of KAL.
Table 8.6 Exclusive prototypical stimulus sentences
Teachers Only Linguists Only
Category Sent. Example Sent. Example
Tense Ä There are a lot of jobs which are needed in the future
Passive ß I was caught 3 fishs German
Interference
C, E, I They did not want that she become queen
Cohesion L, M This is one reason because I
do not become a politician
Concord H There happen not so much
things
Determiner Y The sports don’t play an
important role
Pronouns D, T The state has his own
problems
Verb Prob. F, Ö, U,
V, X, ß
You make your own experiences later
8.2.4.5 Summary
The data in this study do not support the hypothesis that teachers develop a rich network of KAL for teaching. The qualitative analysis found that the teachers’ responses were very similar to the other groups’ responses. The categories for all the respondents were mainly different variations of parts of speech and other language categories. There were only a few cases where categories pointed to specific teaching activities. The qualitative data supported this view. The teachers and the linguists made about the same number of categories in general and the teachers and the control group made “Teacher” categories at approximately the same rate. The analysis of prototypical sentences indicated that linguists seem to have organized KAL, but not teachers. The data from this study further support the findings in the first study that these teachers have not acquired a practice-specific network of KAL similar to the knowledge base of other expert practitioners.