Driving relevance intervention research forward

In summary, relevance interventions have been shown to be an effective tool to enhance students’ value beliefs and—partially—improve grades when conducted in the classroom, especially when students have been asked to write about how science-related subjects are relevant to their personal lives. In addition, attempts have been made to investigate the processes underlying the intervention effects, indicating that factors such as quality and quantity of personal connections could be important for relevance interventions to work. However, further research is needed to advance research on relevance interventions so that findings can be used to improve educational practice and psychological theories and eventually enter educational practice.

FOSTERING STUDENTS’ COMPETENCE BELIEFS AND VALUE BELIEFS IN MATHEMATICS

37 First, successful relevance intervention approaches need to be replicated in different settings with different populations. To scale up psychological experiments in education, Walton (2014) recommended they be replicated within and adapted to diverse educational settings and populations. In fact, it is striking that all of the aforementioned studies were conducted in the United States and have not yet been replicated with students in other nations.

Replication is of paramount scientific importance so as to avoid a distorted picture of intervention effectiveness (cf., Schmiedek, 2016). Although the focus of a range of laboratory experiments has been mathematics-related relevance, almost none of the classroom-based relevance interventions have been conducted in the subject of mathematics (see Woolley et al., 2013, for an exception). Successful interventions, for example those in which students write about the personal relevance of a topic in a text, still have to be tested with a non-American sample and in the subject of mathematics.

Second, new intervention approaches to conveying relevance need to be developed.

Integrating the provision and the self-generation of relevance information into one intervention approach has been shown to be successful in a laboratory setting (Canning & Harackiewicz, 2015). Acee and Weinstein (2010) included a reading passage on the relevance of learning statistics and a subsequent writing activity to have students generate their personal ideas about relevance in a comprehensive value intervention. However, such a combined approach has not yet been investigated in a classroom setting for targeted relevance interventions. In addition, further approaches to conveying the personal relevance of topics need to be tested and compared with conventional approaches to find out which relevance interventions have the strongest and most sustained effects. In fact, a relevance intervention targeting secondary school students’ parents has shown that mailing information about the usefulness of STEM courses (through brochures on a Web site) to students’ parents caused secondary school students to continue taking mathematics and science courses at secondary school for a longer time than students whose parents had not received the respective information (Harackiewicz, Rozek, Hulleman, & Hyde, 2012). Just as the parents communicated their beliefs about the relevance of mathematics or science to their kids, peers or young adults could act as role models for secondary school students and convey the usefulness of mathematical knowledge by describing situations in which they needed mathematical skills.

Third, relevance interventions need to be adapted to students’ genuine classroom environment. Previous relevance intervention studies have been conducted mainly at the individual level, assigning students within classes to different conditions. Such an experimental design has the advantage that variation across classes (e.g., resulting from the teaching style of the teacher) can be kept constant and that at the student level, smaller sample sizes are sufficient to reach a reasonable statistical power. However, within-class randomization also has drawbacks: First, the risk of diffusion effects between students in different conditions within a class is high (Craven, Marsh, Debus, & Jayasinghe, 2001); second, the setting does not correspond with students’ genuine classroom setting where students within classes are typically

INTRODUCTION AND THEORETICAL FRAMEWORK

38

allowed to discuss their common learning experiences. Conducting relevance interventions at the class level opens new possibilities such as initiating class discussions about the relevance of mathematics or science.

Fourth, short- and long-term effects on neglected outcomes need to be investigated.

Wise interventions target recurring psychological processes in order to bring about lasting change (Walton, 2014). It is unclear how students’ motivation and achievement develops once the intervention is removed, as there have been few follow-up studies. Until now, classroom-based relevance interventions have targeted mostly task-related relevance (i.e., writing about the usefulness of a specific topic), but an effective method to have students make personal connections on their own on a regular basis (i.e., for further course topics) has not yet been developed (cf., Hulleman et al., 2017). Targeting students’ general, domain-specific relevance beliefs (instead of task-specific beliefs) could be another way to bring about lasting change. If beliefs about the present and future relevance of a domain are enhanced, this change should affect students’ motivation over and above the topic actually addressed in class. However, to be able to make claims about the sustainability of intervention effects, follow-up studies are needed in which evaluation is made of the effects not only on students’ achievement (e.g., Hulleman et al., 2017; Woolley et al., 2013), but on a range of important outcomes. Behavioral measures (e.g., effort), motivational measures (e.g., competence beliefs), and test-based achievement (instead of grades, which measure more than mere achievement, e.g., McMillan, 2001) have been understudied in prior relevance intervention research. Furthermore, investigation into the effects of relevance interventions on both subject-specific competence beliefs and task-specific competence beliefs is needed to understand better the nature of the relationship between competence beliefs and value beliefs (see 1.3.2).

Finally, in-depth fidelity analyses need to be conducted to unravel the processes mediating the effects of relevance interventions. To be able to make precise adaptations of relevance interventions to different settings and populations, and eventually to pave the way for relevance interventions to enter educational practice, more knowledge is needed about how relevance interventions work, why they sometimes do not work, or why they work for certain students only (e.g., Cohen & Loewenberg Ball, 2007; Murrah et al., 2017; Walton, 2014; Yeager

& Walton, 2011). In previous intervention research assumptions were made about the importance of different elements of students’ responsiveness (e.g., personal connections, cognitive involvement) to the writing activities (e.g., Harackiewicz et al., 2016; Hulleman &

Cordray, 2009). However, no comprehensive analysis has been conducted of students’

responsiveness to the interventions, including the assessment of several indicators of responsiveness, their predictability from students’ individual characteristics, and both descriptive and causal approaches to analyzing their contribution to the intervention effects.

More precisely, identifying individual characteristics that predict students’ responsiveness is necessary to determine possible differential intervention effects and to find ways of adapting the intervention material to the needs of less responsive students, thereby potentially increasing the

FOSTERING STUDENTS’ COMPETENCE BELIEFS AND VALUE BELIEFS IN MATHEMATICS

39 intervention effects (Nelson et al., 2012). Previous research on the importance of students’

responsiveness for the effectiveness of relevance interventions has been descriptive (e.g., Hulleman & Cordray, 2009), and attempts to manipulate experimentally single indicators of responsiveness (e.g., connection frequency) have failed (Hulleman et al., 2017). Including descriptive information on single indicators of responsiveness and conducting in-depth analyses of causal effects based on students’ responsiveness to the writing activities is needed to understand better how single indicators and different degrees of students’ responsiveness actually matter for the intervention effects.

40

3