Retrospective Panel Designs

Figure 20: Retrospective Panel Design which „looks back in time“

Menard describes a retrospective panel design as being “identical to a prospec-tive panel design in every respect except the number of times data collection actually takes place and the length of the recall period required of respondents”

(Menard, 2008). The first aspect, the number of times data collection takes place, refers to the fact that a retrospective panel design collects data only once, but does so retrospectively for two or more periods in the past. The se-cond aspect, the length of the recall period, refers to the fact that in principle every interview/survey that asks a participant to report on certain events is in-herently retrospective. However, a retrospective panel design is fundamentally different, as it systematically asks participants to report on multiple past events that they experienced or observed at different distinct points in time.

Retrospective panel designs offer a variety of advantages over prospective panel designs. First, there is no danger of panel attrition, as participants either take part in the study or do not. Second, the costs of such a study are much more predictable and attractively lower, again because only one data-gathering wave is needed. Third, as Karaponas et al. argue in certain cases experiences reported from memory can be of more interest than the actual experience itself,

t1 t-1

t-2 t-3

t-4

as “these memories (1) will guide future behavior of the individual and (2) will be communicated to others” (Karapanos, Martens, & Hassenzahl, 2009).

However, there are also certain drawbacks to consider. First and foremost, ret-rospective panel designs can only rely on indirect data-gathering techniques, such as inquiry or reconstruction techniques, but obviously not on direct obser-vation or measurement of performance at different points in time. This focuses the value of retrospective designs toward research questions that show interest in opinions, feelings, and the reporting of events or situations. Considering that we are interested in analyzing the learning process for a new interactive sys-tem, we would have to rely on participants’ reports of their feelings, the usability issues encountered, etc., and how these changed over time, but we would be unable to actually measure any of these directly. Second, such designs intro-duce a memory bias, as participants must report on events that often occurred weeks, months, or even years in the past. Menard (Menard, 2002, p. 44) states that such designs work better with salient events than with attitudes or other psychological data, as events are more objective. In case of attitudes, partici-pants tend to create a consistent life story and align earlier attitudes with their current ones. Through the act of retrospection, they reflect upon their life or the period in question and reinterpret their memories. Whenever possible, short term retrospection should be preferred. Nevertheless, Taris states that “a pro-spective longitudinal design will virtually always result in better (more reliable and more accurate) data than a retrospective design” (Taris, 2000).

Considerable effort has been conducted to improve this memory bias, such as the iScale approach (Karapanos, Martens, & Hassenzahl, 2009), the Day Re-construction Method (Kahnemann, Krueger, Schkade, Schwarz, & Stone, 2004), and the CORPUS interview technique (von Wilamowitz-Moellendorf, Hassenzahl, & Platz, 2007). All of these techniques involve a certain explicit construction process which is meant to help participants to remember their ac-tual experiences and correctly place them in the time dimension.

iScale for example aims at eliciting the longitudinal user experience by asking the users to sketch these on a simple time-scale with a UX dimension on the

y-scale. Important events could be marked and annotated and the authors even developed an electronic tool that implements this technique. Figure 21 shows an example of such a drawing with the ease of use on the y-axis and time on the x-axis. The numbers and annotations mark certain events in time and allow the researchers to understand these events and their relationship with the over-all usage experience.

Figure 21: Example graphs from a iScale study with hand-sketched graphs (taken from (Karapanos, Martens, & Hassenzahl, 2009)).

Another often overlooked drawback pointed out by Menard is the inherent bias of participant selection, or sampling bias, as he calls it (Menard, 2002). In the most extreme case, participants who have died before the retrospective study takes place are completely excluded from taking part in such a study, even though they could have been part of a prospective panel design for the first few data-gathering waves. While this situation is rather uncommon in HCI research, the problem still applies to a lesser extent in other situations. Consider a retro-spective study that tries to identify how the experiences of using a certain mo-bile phone change over time. Naturally, one would try to find participants who are owners of this mobile phone; however, this would implicitly exclude all those who have stopped using the device and e.g. changed to a competing product.

In this specific case, it is possible to also search for participants who have owned the specific device in the past to avoid this bias. However, it is important to think about the kind of participants that are naturally excluded from the study if it is mistakenly framed in this intuitive but incorrect fashion.

2.3.6 Relationship between Research Questions and Research