Assessing and analyzing results

Performance test’s results for the 3D-on-2D variant shows some minor lag issues while mov-ing along the timeline. Delays on rendermov-ing time are negligible.

The overall performance of the VR variant was smooth except some minor lags in the cursor animation and potential of an overheat issue if used for an extended period of time. Perfor-mance took a considerable negative impact when the mobile browser had to render a large

number of 3D objects and the device was overheated quickly. For more capable devices however, performance is expected to vastly improve. The VR variant performed and ren-dered best on the Google Chrome browser, but encountered various rendering issues on other browsers such as mobile Safari, due to the fact that the WebVR standard being still in early stages, and thus not uniformly supported cross-browsers.

To assess the results of the UX test, we shall first begin with the analysis of individual ad-jectives | cards and the frequency with which they have been selected. The sample pool consists of 66.7% of the 18-29, and 33.3% of the 30-39 age group, and is divided 50/50 between the novice and beginner level of familiarity to the Virtual Reality environment. To get a quick overview of the impressions at first glance, the frequencies of the selected cards are visualized using word clouds to present the quantitative outcome, as shown in Figure49 and Figure50. The colors of the words reveal that the initial impressions of the 3D tube of the VR variant and the guided tour along the timeline with narrations received more negative responses than other parts of the visualization, and the size of the word tell us that the 3D stacked bar chart, the 3D tube, the slow timeline tour and the overall VR visualization were seen as interesting, informative, logical, and useful. The 3D bar chart was appraised as quick to understand, the showing of all the tubes at once was seen as complex and clumsy, while the VR variant was assessed as more innovative and intuitive than the 3D-on-2D coun-terpart. It is also easy to recognize that the quick tour without narrations was appraised with the most positive impressions.

While word clouds provide a quick overview and enable rapid assessment, they fail to com-municate results in a clear manner that would allow for a more in-depth comparison. Classi-cal charts would enable for a more precise observation of the differences between selected cards within an individual visualization or in comparison to another variant. Figure45shows the selection of impression cards for the VR variant in comparison to the 3D-on-2D variant.

The chart tells us that participants in the study recognized the VR variant as significantly more fun, innovative and intuitive, besides being easier to use and more logical. On the other hand, it is also possible to see that more than half of the participants also thinks it is somewhat more time-consuming, while a few opinions show it is more clumsy and somewhat useless.

Although the assessment of the selection of individual cards is the more common approach and in most cases, sufficient for result analysis, some other types of exploration can also be quite useful and informative. For example the overall number of selected cards for individual tasks can be an indicator of the user’s attitude toward that part of the visualization and also how engaging it is. The figures in Table 5 make it clear that the VR visualization is most engaging as a whole, while the participants had the most impressions in Task 1 and 3.

The ratio of positive and negative cards as shown in Table6even better reflects the overall results of the UX test as Task 1, 3, 4 and 6 gathered the highest rates of positive impressions, while Task 2 and 5 had a significantly higher percentage of negative cards. The fact that the

Figure 45: Reaction cards selected for the VR variant in comparison to the 3D-on-2D variant.

overall impression task received over 80% of positive cards therefore tells us that the VR prototype of the visualization design evoked also positive UX.

We also need to assess the results in term of the five dimensions into which the reaction cards are categorized. The radar chart provides the most suitable visualization tool for this type of analysis. For graphical presentation, positive and negative impressions are separated a and to simplify the analysis, it is sufficient to only visualize positive aspects, as negative cards basically form a mirror image of their positive counterparts. Figure51maps the values collected from the reaction cards to the five dimensions. This presentation provides us with a higher overview of the UX test results, for example, it can be observed that Task 5 (show-ing all 3D tubes) scored relatively low on ease of use and usefulness, but better in term of engagement and appeal. This chart also gives us a good idea of which parts of the visu-alization prototype scored an overall lower UX, based on the values achieved on individual dimensions.

Figure 52 reveals another interesting aspect in term of the shape of the graph. The two charts show that while Task 1, 2, 5 and the overall visualization created a more balanced

Task 1 Task 2 Task 3 Task 4 Task 5 Task 6

66 57 65 58 54 73

Table 5: Total number of selected cards per task.

Task 1 Task 2 Task 3 Task 4 Task 5 Task 6

+ 88% 75.44% 86.15% 100% 68.52% 84.93%

- 12% 24.56% 13.85% 0% 31.48% 15.07%

Table 6: Ratio of selected positive and negative cards.

shape and thus hint at a more all-round UX, Task 3 and 4 formed a skewed shape which indicates that in those tasks, the five UX dimensions are unevenly supported.

Lastly, the radar charts also enable the calculation of their surface areas, the result of which gives a mathematical expression to the overall UX. This method is defined in (Mosley and Mayer, 1999) and known as Surface Measure of Overall Performance (SMOP) and typically used in benchmarking analysis. Using the formula:

((P1∗P2) + (P2∗P3) + (P3∗P4) +· · ·+ (Pn∗P1))∗sin(360/n)/2

where P is the data point on the axis of the radar chart and n is the number of axes, the surface area of the radar graphs can be calculated and shown in Table7. The results of the surface area calculation clearly indicate that the parts of the visualization corresponding to Task 1 and Task 4, by this measure, provide the best UX.

Task 1 Task 2 Task 3 Task 4 Task 5 318.6 175.5 299.6 318.6 132.2

Table 7: Surface area of radar graphs.