Katharina Hagenfeld
3. Assessing interlanguage development with Rapid Profile and Autoprofiling
This chapter briefly introduces Processability Theory and its diagnostic tools Rapid profile and Autoprofiling. It will end with listing advantages of these testing procedures concerning a possible enhancement of quality testing criteria in the language proficiency ratings.
Processability Theory (PT) (Pienemann 1998, 2005) is a psycholinguistic the-ory of L2 development that predicts a universal developmental hierarchy for the acquisition of a second language. The hierarchy is spelled out based on the archi-tecture of the human language processor as modeled by Levelt’s blueprint for the speaker (1989).5 PT’s core assumption is that a linguistic structure can only be produced and consequently acquired if the current state of the language proces-sor is capable of processing the respective linguistic form (Pienemann 2007: 137).
Initial psychological constraints account for a cumulative and successive acqui-sition process that is implicationally related. An implicational order infers that a later structure and procedure “implies the presence of an earlier structure”
( Pienemann 2011: 51; Pienemann & Keßler 2011). Two key mechanisms are crucial in this regard: (a) feature unification and (b) specific mapping processes as modeled in Lexical-Functional Grammar (Bresnan 2001).6 Feature unifica-tion accounts for the matching of grammatical features as produced by Levelt’s
5. Due to the scope of this paper, a detailed description of Levelt’s model of message generation will not be given. For further information, please see (1989).
6. Lexical Functional Grammar is a formal theory of grammar that assumes three levels of linguistic representation to be linked by mapping principles. Argument structure contains the meaning to be expressed (verb and its arguments) that is to be mapped onto the surface form as represented in constituent structure. Functional structure in which grammatical features are encoded links argument and constituent structure. Please see Bresnan (2001) and Lenzing (2013) for more information.
Psychometric approaches to language testing and linguistic profiling 141 processing components (Levelt 1989: 9). LFG’s mapping principles allow for the spelling out of the developmental hierarchy for a variety of typologically diverse languages such as Chinese, English, Swedish, etc. PT therefore, does not use the term proficiency in L2 but development. L2 development is defined as those gram-matical features that are captured in the hierarchy. The hierarchy for English as a second language is spelled out as follows:
Stage Processing
procedures Phenomena Examples 6 Subordinate clause -
procedure Cancel Aux-2nd I wonder what he wants.
5 S-procedure Neg/Aux-2nd-?
Aux-2nd-?
3sg-s
Why didn’t you tell me? Why can’t she come? Why did she eat that?
What will you do?
Peter likes bananas.
4 VP- procedure Copula S (x) Wh-copula S (x)
Where she went? What you want?
Today he stay here.
I show you my garden. This is your pencil.
Mary called him.
2 Category procedure S neg V(O) SVOSVO-Question
Figure 2. PT hierarchy for English as a L2, adapted and modified from Pienemann (2005: 24) Figure 2 depicts the universal processing procedures at the left hand side along the structural linguistic realization of these exemplified with illustrations. Stage one represents the lowest stage at which words and formulaic utterances can be retrieved from the mental lexicon (as modelled in Levelt (1989). The higher the stage, the more productive the learner becomes in the target language.
142 Katharina Hagenfeld
In order to capture and use the power of PT to predict the course of L2 devel-opment, diagnostic tools to determine the current stage of acquisition of L2 learn-ers were developed. A benefit of knowing the current stage of learner development is to enable language instructors to provide targeted instruction and to pick the learners up where they are.
Linguistic Profiling is based on theoretical work by Crystal et al. (1982) and Clahsen (1985) and follows the profiling approach by Crystal, Fletcher, and Garman (1976) in the domain of language disorders7 and its adoption and modi-fication for German L2 acquisition by Clahsen (1985). For a profile analysis an interview is conducted, a full transcription of the interview is done on which a careful analysis of the sample is based (Pienemann & Mackey 1993: 24). The pro-file approach as used with LARSP is in line with the view of language assessment taken by Pienemann (1998) as it is
(a) descriptive, (b) developmental, and (c) interactive. The first refers to the descriptive categories provided by the procedure; the second, to the developmental schedule of these categories; and the third, to the method of data collection: the spontaneous speech gathered in unstructured conversations. […] Descriptive criteria are objective; developmental criteria are psychologically plausible, and interactive criteria are based on natural language use.
Pienemann, Johnston, & Brindley (1988: 231) For a language profile, natural oral speech data is elicited and scrutinized using distributional analysis. Distributional analysis as used by Pienemann (1998: 139) allows for determining “which contexts or even which lexical items are related to which particular interlanguage rules.” This way, idiosyncratic and formulaic use of the target language can be ruled out (please see the following section on the emergence criterion for further explanation). Since a careful and fine-grained dis-tributional is quite time-consuming and hardly feasible in ESL contexts, a rapid version for the allocation of interlanguage development was established, i.e. Rapid Profile (RP). RP (Mackey, Pienemann, & Thornton 1991; Pienemann & Mackey 1993; Keßler 2006, 2008) is a computer-assisted screening procedure operated by trained linguistic profilers (cf. Keßler 2006). It is a short-hand version of the origi-nal linguistic profile. In RP the profiler uses communicative tasks that trigger the production of a specific linguistic structure found in the Processability hierarchy.
The tasks focus only on those linguistic items that are crucial for determining the
7. Language Assessment Remediation and Screening Procedure (LARSP) is a screening pro-cedure to allocate learner language in terms of grammatical disability. It has been widely used up to date, mainly by speech therapists and language researchers. For more information, see Crystal, Fletcher, & Garman (1976), Crystal (1982).
Psychometric approaches to language testing and linguistic profiling 143 developmental stage of a learner (cf. Pienemann & Mackey 1993: 25). They usually contain an information gap to make the subjects produce specific linguistic struc-tures without noticing them. The 3rd person singular-s, for example is elicited through a habitual action task that shows pictures of the daily routine of a fictive person. In order to gain a dense data set, at least three different tasks aiming at different structures in the Processability hierarchy need to be used. During the production phase of the learner, the profiler uses the computer interface to check those buttons that relate to the structure produced. Figure 3 shows the RP inter-face with the boxes for each structure in the hierarchy.
Figure 3. Rapid Profile 4.0 user interface
The structures are subdivided into syntactic phenomena on the top and mor-phological phenomena on the bottom of the interface. If the learner produces a verb in an obligatory context along with a morphological feature such as the past-ed, the profiler clicks on the plus under the headline verb. Should the learner fail to attach a past-ed in an obligatory context, the profiler would check the minus-box. The program computes the developmental stage in checking the data typed in against standard learner language according to the emergence criterion (see the next section for further information). Rapid Profile gives detailed feedback not only on the developmental stage but morphology and syntax. The very recent development of Autoprofiling (AP) (Pienemann, Lin, &
Chung 2009; Lin 2012) further simplifies this procedure by working analogous to RP. AP is an online screening procedure that operates fully automatically.
With AP, the learner simply types in his/her answers to the tasks into an input field. The interlanguage sample is calculated in comparison to a small corpus
144 Katharina Hagenfeld
that is embedded in the program (Lin 2012). The sample is conducted under a time constraint that rules out the usage of declarative monitoring.8 The pro-gram works similar to a common RP analysis but is accessible from everywhere and at anytime. As opposed to RP, AP works with written input. This how-ever should not have any influence on the developmental level of the learners since the mode-steadiness hypothesis (Plesser 2008) predicts that interlan-guage development remains within the concept of hypothesis space even across mode-barrier boundaries. Håkansson and Norrby (2006) underline Plesser’s findings in comparing written and spoken L2 Swedish. Their learners followed the PT hierarchy on both modes with a tendency for learners to be one level ahead in written production.
Advantages of Rapid Profile lie in the computer-assisted nature of the pro-gram that compares standard patterns of development with a learner’s inter-language sample (Keßler 2006). Thus, the program scores high in objectivity.
Trained profilers are able to elicit a profile with high inter-rater-reliability (Keßler 2006: 241). This is why the use of RP allows for accommodating reliable and valid results in only fifteen minutes (Keßler 2006). In his study, Keßler (2006) tested as to whether fifteen minutes were sufficient enough in order to elicit a dense data set.
His results showed that “[…] the data elicitation took an average of 12.5 minutes and ranged between seven and 17 minutes” (Keßler & Plesser 2011: 214) with sufficient data density.
While Rapid Profile establishes high standards for language testing in terms of rapidness and reliability, a disadvantage is the profiler-dependent usage of the program. Autoprofiling by Lin (2012) has the potential to overcome this limitation since it operates fully automatically. Lin (2012) showed that there is 99.0% accor-dance of AP results compared to RP results. To recapitulate, LP scores sharply high not only in measurement and testing criteria as defined by Bachman (1990, 2004), Rasinger (2008) or Neuendorf (2002) but includes a very detailed inter-language and grammatical feedback. Thus LP is able to provide hands-on feed-back to the language learner with clear indications as to what s/he is able to do at this specific point in her/his development as well as what is learnable next.9 This predictive power that underlies the results of LP further enables teachers to
. The time constraint is embedded to avoid that a learner goes back to her/his written work and changes it terms of style and accuracy as this would lead to a distortion of the profile. PT assumes procedural knowledge to be more important in the production of a second language than declarative knowledge. For further information, please see Plesser (2008), Ellis (2005, 2007).
9. This is one of the criteria that Brindley (1998: 117) considers to be crucial in language testing.
Psychometric approaches to language testing and linguistic profiling 145 internally differentiate their students according to their developmental levels and give respective instruction. The following section further elaborates on the predic-tive power of PT.
3.1 The emergence criterion in Rapid Profile and Autoprofiling
The emergence criterion (EC) (Meisel, Clahsen, & Pienemann 1981; Pienemann 1998; Pallotti 2007) assumes that as soon as a grammatical structure appears in the interlanguage of a learner, the structure is assumed to be acquired (Keßler 2006: 147; Pienemann & Keßler 2011). Thus, the production of a linguistic struc-ture defines “[…] the beginning of an acquisition process, and focusing on the start of this process will allow the researcher to reveal more about the rest of this process.” (Pienemann 1998: 138). Thus, RP and AP need three incidences of syntactical and morphological and lexical variation to assume a structure to be acquired. Morphological and lexical variation is exemplified in Figure 4 with the help of three verbs and three morphemes below.
Lexical variation Morphological variation go
+ play + talk
-ing + -ed + -(e)s Figure 4. Illustration of emergence criterion
Thus, to predict that the third-person singular-s is de facto acquired, it would need to be attached to all verbs above and the learner would have to use the different inflectional endings as well. This way, a mere storage of stem and affix as a chunk in the mental lexicon (as modelled in Levelt 1989) can be ruled out as the learner has to use both, ending and verb, creatively. The emergence criterion allows one to pinpoint the acquisition of an underlying interlanguage structure in a direct manner. Using the EC as the point of acquisition, no third party has to judge or rate whether a feature might have been attained. Unlike with rating scales, the usage of introspective or retrospective means (such as questionnaires or thinking aloud protocols) on the sides of learner is not necessary either.
In making a case for the integration of linguistic profiling into the CEFR, the emergence criterion as an objective means to indicate language development
146 Katharina Hagenfeld
strengthens the validity of PT and its assessment tools and is thus considered crucial in this context.
In the following section, the study and its components along the results are outlined.
4. The study
This cross-sectional study is a pilot study which was conducted in order to make inferences about a possible integration of linguistic profiling into the Framework of References for Languages. One has to note however, that PT takes a modular approach to LA whereas the CEFR covers language and its acquisition holistically.
4.1 Aims and research questions
The overall goal of this study is to investigate whether LP can be used as a comple-mentary assessment tool to the CEFR in order to provide a more objective, reli-able and accurate feedback for the testees. Thus, my aims are the following: (a) In order to hypothesize an integration of linguistic profiling into the CEFR, it must be ascertained that there are correspondences between the CEFR levels and the developmental stages as predicted by PT. This study therefore takes another look at the findings by Lenzing and Plesser (2010) who piloted the quest for correspon-dences. In a pilot study, they found that PT stage three relates to CEFR level A1, stage four to A1 and B1, stage five to B1, B2 and C1 and stage six to C1 as can be seen in Figure 5.
Rapid Profile CEFR Level
Stage 1 Below A1
Stage 2 Below A1 and A1
Stage 3 A1
Stage 4 A1, B2
Stage 5 B1, B2, C1
Stage 6 C1
Figure 5. CEFR and Rapid Profile correspondences, Lenzing & Plesser (2010)
The present study reconsiders their findings with further data. Once correspon-dences are laid out, it is believed that a combination of LP based on PT and the CEFR can cover many aspects of language proficiency and development with
Psychometric approaches to language testing and linguistic profiling 147 enhanced results in objective feedback and reliability. RP and AP’s beneficial backwash then allows teachers to provide learners with materials based on the predictions of PT to help them progress in their interlanguage development.
Another aim of this paper is to examine the relationship between Rapid Profile and Autoprofile. As mentioned in Section 3, Autoprofile is assumed to be (b) more feasible in large-scale assessment settings than Rapid Profile is due to the fully automatic nature of the program. One has to however, go a step back and examine whether Autoprofile shares RP’s benefits in terms of (c) reliable diagnos-tic outcome and whether AP may even exceed RP in the time span needed for the assessment. A faster assessment is generally seen as more feasible. Without mak-ing sure that the latest addition to the PT formula is as reliable as its predecessor, making claims about its integration into the CEFR is superfluous. This is why RP and AP feedback will be compared in terms of the developmental stage and the time allotment both programs compute.
These aims generate the following research questions:
a. Are there are correspondences between CEFR levels and PT stages?
b. Is AP more feasible than RP in rating settings due to its profiler-independence and rapidness?
c. Are RP and AP equally reliable?
I hypothesize that there is indeed correspondence between CEFR levels and PT stages as indicated by the findings by Lenzing & Plesser (2010). Due to the nature of AP, I further hypothesize that AP scores higher than RP in time allotment and infer that both programs are equally reliable in terms of feedback. In order to test these claims, the data were elicited as follows.
4.2 Data and methodology
For the study, speech samples of nine university students were collected out of which three were male and six female. The students attended English courses at different CEFR levels from B1 to C2. Their professions differ widely, ranging from mechanical engineers (three male students), sport science students (two female students), business students (two female students) to a teacher trainee (one female student). A biodata-questionnaire elicited the reasons why the participants took part in English-courses. 90% of them wrote they wanted to refresh their English and 10% take the course as a preparation for their future occupation. Prior to the courses, the students either took the Oxford Placement Test (OPT) online or they were rated by trained raters. The OPT shows high correspondence between the score the participants achieved and the CEFR levels. The OPT scale and the according recommended courses can be viewed from the appendix in this volume.
Every learner participating in this study was thus rated according to the CEFR
14 Katharina Hagenfeld
and profiled with Rapid Profile as well as Autoprofiling in order to compare their CEFR level and PT stage.
Once the participants completed the OPT, they attended the respective lan-guage course. These courses run on a weekly basis. At the end of the course, a test following CEFR criteria is taken which shows whether the students fulfil the requirements. In working with these learners, it was assured that they were all suf-ficiently rated according to CEFR standards.
The RP profile was conducted with two (and three participants at level B1) at each CEFR level performing communicative tasks in pairs. The participants were audio-recorded and the recordings were transcribed. The RP analysis was con-ducted by a trained linguistic profiler and checked against the transcriptions. After the RP profile, the students were briefly introduced to the AP interface. They were given time to familiarize themselves with the program and had the opportunity to ask technical questions. Since AP operates fully online, the software can be used anywhere and anytime without a profiler being present. Thus, the students were not able to ask questions during the procedure. Both, RP and AP work with the same task design that only differs in subject matter. The communicative tasks that were used provide natural obligatory contexts for specific linguistic structures to be found in the PT hierarchy and are in line with the standards set by Mackey, Pienemann, & Thornton (1991), Pienemann & Mackey (1993).
Rapid Profile Autoprofile
Figure 6. Tasks and the linguistic structures they trigger10
10. This is a summary based on work by Meisel, Clahsen, & Pienemann (1981); Mackey, Pienemann, & Thornton (1991); Keßler (2006), (2008); Lenzing (2010); Plesser (2011).
Psychometric approaches to language testing and linguistic profiling 149 Figure 6 shows the tasks used in RP and AP and those linguistic forms that are triggered due to the task design. With RP, three tasks are used in which the pro-filer has to guide and moderate the conversation in order to gain enough data. In AP, four tasks are administered in which two aim at declarative sentences (picture description and habitual action task) whereas the two interviews trigger interroga-tives. In this way, a most holistic profile in terms of syntactical and morphological features that are captured in PT is established.
The time it took the participants to complete the AP analysis was determined and compared to the length of the audio-recording for RP. For each subject, the CEFR level and developmental stage elicited with RP and AP were recorded. The results are as follows.
The time it took the participants to complete the AP analysis was determined and compared to the length of the audio-recording for RP. For each subject, the CEFR level and developmental stage elicited with RP and AP were recorded. The results are as follows.