Statistical inference implying the act to generalise from the sample to the population (Tredoux, 2009, pp. 2-17), to ensure external validity can be justified on the size of the sample, general representativeness of the population in terms of the curriculum content, study programmes, qualifications and guiding policies. Internal validity, referring to the research design and measuring tools, was obtained by implementing pre-tests to validate the COMET test tasks.
128
All the questionnaires were administered in person to groups of students as written questionnaires. This approach allowed for the completion of surveys once started as well as clarification where needed (Cozby, 2005, pp. 134-135). Rating scales applied in conjunction with COMET measurement for context questionnaires include Binary scales assuming one or two possible values such as yes or no, true or false and Likert scales indicating respondent statements to an extent of agreement or disagreement (Bhattacherjee, 2012, pp. 45-49).
The following measuring tools were applied:
4.4.1 COMET occupational competence assessment and development open-ended test tasks.
Subject experts developed the open-ended test tasks. A Rater Training Seminar followed. Test validation was done during a COMET pre-test. This pre-test or pilot testing assisted in detecting possible problems and it ensured that test tasks selected for the main COMET large-scale assessment were reliable and valid measures of assessing holistic occupational competence (Bhattacherjee, 2012, p. 23). Four test tasks per occupation (Electrical, Millwright, Mechatronics and Welding) were selected. Occupational Competence was measured on the three dimensional COMET occupational competence and commitment model. A high degree of consistency also referred to as inter-rater reliability, in the rating of solutions is a prerequisite for open-ended test tasks. The 40-item rater responses were validated by means of an exploratory factoring analysis based on the correlations among the 40 items. A typical test task comprises a realistic narrative of a problem, typical and representative of the occupation at stake, to be solved by means of professional problem solving, using the eight COMET criteria.
The degree of complexity must allow for the assessment of contextual understanding. The grading of the test outcomes is ability based, making it possible to differentiate among test takers according to levels of solutions presented as being functional, procedural or holistic of nature (Rauner, 2010).
4.4.2. COMET Vocational Identity & Occupational Commitment Questionnaire
This tool explores the participants’ attitudes and values towards their profession and training.
It comprises of five sections: 1. Personal data and general questions regarding their training company; 2. Personal opinion of students regarding their professions; 3. Company based
129
Vocational training; 4. TVET College State of affairs and 5. Communication and cooperative Teamwork.
4.4.3. COMET Motivational Questionnaire
This questionnaire measures the levels of motivation applied in test task completion as well as experience of the relevance thereof for TVET. Test tasks complexity, Time spent on task, Student commitment and interest in the test tasks and the student’s overall sense of approval for this type of assessment are explored.
4.4.4. COMET Practical Test Tasks and Interviews
The close partnership between workplaces and institutions of learning is a major priority for DHET and the Minister of Higher Education and Training is delighted to see it being endorsed in the motto of the Skills Handbook 2015/2016: Every workplace a learning space (Stuart, 2015). It is therefore imperative to explore the impact of COMET on the assessment and development of occupational competence in a practical setting. One TVET College as well as one Industry Training Academy exposed to COMET and DSAP/Apprenticeship participated.
There is an increasing appeal for adaptability to new demands, responsibilities and technologies in the workplace. The traditional approaches that focus narrowly on the mode of production are of limited value. TVET should aim to provide a sound foundation in particular skills while inculcating the skills needed to respond to new production modes (Government Gazette No. 38796, Vol. 599, 2015). This investigation consisted of two processes- Firstly the COMET Practical task executed by students and secondly, an interview with the students on COMET dimensions covered in the Practical task. Two COMET Raters rated the Practical task.
130 4.4.5. Feedback Questionnaire.
This Questionnaire comprises of two sections of 15 items each, measuring the impact of assessment feedback on learning and the development of occupational competence. Section A measures Assessment Feedback in the current TVET Curriculum and section B, COMET Assessment Feedback. TVET staff (N=18) involved in COMET assessment and Rater training completed the questionnaire. Raters reported on Assessment Feedback criteria applying a 4-point Likert scale ranging from Strongly Disagree to Strongly Agree.
4.4.6. Social Competence Survey
Senior students from two TVET Colleges (N= 20 + N= 20) as well as one Industry Training Academy (N=22), exposed to COMET and DSAP/Apprenticeship completed this survey.
TVET students (self-rating) and TVET staff reported on student social competence on the same instrument measuring eight Social Competence dimensions, defined by four criteria each. An average rating was calculated. Students and Staff reported on the 32-criteria measurements with a 4-point Likert scale from 1(Never) to 4(Always).
4.4.7. Focus Group Semi-structured Interviews
The interview as a research method is seen as the most prominent data collection tool in qualitative research. It is a powerful technique to understand and assess people’s perceptions, meanings, definitions of situations and the construction of reality (Punch, 2009, pp. 144-148).
There are many forms of interviews. For this study, focused group, semi-structured interviews were deemed the most suitable method to use in conjunction with the COMET quantitative assessment. Questions were open ended and posed to the group allowing for group interaction where certain comments triggered a variety of responses (Cozby, 2005, pp. 136-137). The group interaction yielded valuable information to improve sense making of behavior and performance measured in the quantitative research. TVET students who completed the Social Competence questionnaire at the three sites also participated in the interviews - (TVET 1- N=
131
20 + TVET 2- N= 20 + Industry Training Academy N= 22). The researcher served as a facilitator, moderator and mediator while nine open-ended questions alluding to the impact of TVET on improving quality of life, differences between Curriculum assessment and COMET assessment, experiences of the DSAP, Student readiness for the workplace, Feedback process after assessments and open comments were discussed, recorded and transcribed. Main emerging themes were identified.
COMET open-ended test tasks and context questionnaires will enable the researcher to collect a significant amount of in-depth data to respond to the research question. Focus Group Interviews using open-ended questions posed to the group allowed for dynamic group interaction. Valuable supportive and clarifying data can be derived from such interviews.
4.5 DATA ANALYSIS