Strengths and Limitations of the Present Dissertation

In addition to these implications for research and practice, some limitations have to be discussed. In this section, limitations are listed and discussed and the dissertation’s strengths are elaborated.

Sample

The students of the three studies were participants in a presentation contest for secondary school students who had successfully passed the first round of the contest (i.e., jury evaluations of their application videos). Consequently, a highly selective sample was used. It must be assumed that participants in a contest are not representative for secondary school students in general. A survey focusing students in a higher track found that the participation in scientific contests depends on high intrinsiv competence pursuit, a high competition self-concept and previous contest participation (Blankenburg et al., 2015). Thus, it can be assumed that the participants of the Youth Presents contest are mainly high achievers and highly motivated (see Rebholz, 2018; Stang et al., 2014; Urhahne et al., 2012). The descriptive statistics showed that more than 95 % of the students attended a Gymnasium, the highest track in the German school system. Thus, the generalizability of the study results to other school tracks is limited. However, in developing the TIP, this dissertation did not only focus on the most successful students in the contest. It also included students who were excluded after the second round. Thus, the TIP was tested on a sample of students with different presentation competence levels. Nevertheless, it can be assumed that the variance in presentation performance would be greater in a more representative sample of secondary school students.

Furthermore, this dissertation found treatment effects among a specific sample of highly motivated and high-performing students. This can serves as a starting point for successively extending the sample up to a representative sample.

Study design

The three empirical studies were based on presentation tasks with different standardization levels. While the presentation task at T1 was semi-standardized, the presentation task at T2 was fully standardized. This limitation weakens the study design and represents a methodological limitation for aspects such as the measurement of stability.

However, despite these differences in standardization across measurement points, the degree of standardization within each measurement point remained the same (S. B. Green, 2003).

Consequently, the entire study sample completed the same presentation task at each measurement point. For example, the presentation rooms had the same configurations and the audience consisted of exactly the same number of listeners. Confounding factors were also minimized at each measurement point with respect to the preparation of the presentations. For example, at T2, privacy screens were installed on the tables in the preparation room, and a test instructor supervised the room to ensure that each student prepared individually. This increases the interpretability of the results (G. Brown & Hattie, 2012). A further strength is the construction of the presentation tasks. All of the presentation tasks can be classified as authentic tasks (Chan, 2011; Guariento, 2001). For example, there were analog visual aids as well as digital visual aids. The tasks were related to the real world and similar to presentation-based exams in school. The students had time to prepare for the presentation task and delivered their presentation in front of physically present, real people. The presentation tasks at T1, T3, and T4 were high-stakes tests (Stobart & Eggen, 2012), because students’ performance determined whether they qualified for the next round of the presentation contest. T2 can be classified as a low-stakes test. Finally, because the study only included individual and no group presentations, presentation performance can be clearly attributed to the individual level.

Video ratings as external assessment tool

The assessments of presentation competence via the TIP were based on video-recorded presentations. This method is also likely to have limitations. The video format can influence perception of the observed behavior, resulting in biased assessments (Nagel, 2012). For example, the selection of camera angle (Baranowski & Hecht, 2017) or the video quality can influence the observer’s perceptions (see Watson & Sasse, 1998). In addition, the context forms a central reference frame for interpretation (Curby et al., 2016). In contrast to live impressions, videos do not transmit room temperature, the feeling of an object that is passed to the real audience, or the scent of the people or the room (Nagel, 2012). This dissertation took these influences into account and tried to reduce biases on a technical as well as on an information basis. From a technical standpoint, no editing of the presentations was conducted. The presentations were all recorded from one camera angle. To capture a perspective similar to the live audience, the camera angle was at the height of the physically present audience and centered on the stage where the speaker stood. In addition, context information was provided for the video raters. They received a situation card including three photos of each presentation room before watching the presentation. These room photos were from three different perspectives (the speaker’s perspective, the audience’s perspective, bird’s eye view of the room)

to support the raters’ ability to imagine the physical distances in the room. In addition, the raters were informed about the situation, the presentation task, and the instructions the students had received for their presentation. For example, the students were told to ignore the camera and to deliver their presentation for the audience members. In addition, the video raters were trained to ignore aspects such as the lighting of the room or quality of the video and instead focus on the actual presentation behavior.

TIP in the field of tension of existing instruments

The development, benefits and limitations of the TIP could be described in comparison to well-established instruments such as the Competent Speaker, which has been used in several studies (e.g., Clyde et al., 1994; Gring & Littlejohn, 2000; Smith & Sodano, 2011). The Competent Speaker assesses students’ presentation competence in an objective, reliable and valid way (Morreale et al., 2007). Moreover, the Competent Speaker encompasses 8 items on a three-point Likert-type scale, offering a usable and a manageable procedure to observers. In fact, there is much evidence supporting the use of this instrument in this dissertation as well.

However, there are also several arguments against it. The eight-item Competent Speaker measure does not address all relevant aspects deduced from rhetorical theory. For example, addressing the audience is not part of the instrument. Furthermore, Brown, Leipzig, &

McWherter (1997) criticized the measure’s abstract assessment criteria and called for a more detailed perspective on observable presentation behaviors. In addition, the three-point scale limits opportunities to provide specific feedback.

The goal of this dissertation was to assess presentation competence on a broader basis by focusing on secondary school students’ presentation competence development. The chosen approach was to develop a new instrument based on the strengths and limitations of existing instruments and taking into account the rhetorical perspective on presentation competence.

Thus, the TIP focused on providing a detailed view of presentation competence from a rhetorical basis and a more specific perspective on presentation behavior. This is also in line with the Public Speaking Competency Instrument (PSCS) by Thomson and Rucker (2002) based on 20 items, or the Public Speaking Competence Rubric (PSCR) by Schreiber and colleagues (2012) based on 11 items, both of which took a more detailed view on the construct of presentation competence. However, even these instruments do not cover all of the presentation facets deduced from rhetorical theory. Language use is not part of the instruments by De Grez (2009) or Thomson and Rucker (2002). Furthermore, the instruments’ authors do not argue why specific presentation behaviors are included or excluded. No empirical findings

are provided to explain why some specific presentation behaviors are missing. Consequently, this dissertation took a step back and developed the TIP instrument, which provides arguments for including specific presentation behaviors based on rhetorical theory, and conducted first analyses on reducing the number of items via exploratory factor analysis. In addition, use a three-point Likert-type scale limits the specificity of feedback. The TIP and other instruments (e.g., Herbein, Golle, Tibus, Schiefer et al., 2018; Schreiber et al., 2012; Thomson & Rucker, 2002) use four- or five-point Likert-type scales. Moreover, the TIP scale defines specific indicators and anchors for presentation behavior for each point on the Likert-type scale. In addition, each point of this scale reflect the development of presentation skills. The results of the first examinations of the TIP indicate its added value in comparison to existing instruments.

However, further examinations are required to underpin this tendency. Future research should also include ratings using these established instruments to further analyze the benefits of the TIP.