A.3 Supplementary information
5.5 Model
5.6.2 Quantile regression
In order to shed more light on the question who benefits from the practice, we present subsequently the results of the quantile regression model specified in Equation (5.3).
The covariates inXi are either specified as the union or the intersection of all relevant covariates identified in Section5.6.1. The results are graphically displayed in FiguresD1 toD4, restricted to the practice variables of interest. Confidence intervals of the quantile regression coefficients are based on standard errors calculated using a wild bootstrap procedure with 5,000 replications.
Figure D1 presents the coefficient estimates on the number of midterm tests. In both specifications (a) and (b), the parameters are usually statistically significant with some exceptions for quantiles around 0.8 in (a). Regarding Figure D1a, we can see that stu-dents who are expected to achieve a low number of points in the exam, i.e., the lower quantiles, benefit more from the practice than students who are expected to attain a high number of points anyway. For the lowest quantile, additional practice adds 6 points to the final exam while the effect goes down to roughly 2 points for students in the upper quantile. When using all control variables (see Figure D1b), this pattern is less pronounced. The lower quantile estimates still are as high as before, but the upper quantile estimates vary now around 4 as well. Hence, practicing is helpful in general, and it seems to be the case that weak students benefit even more than otherwise good students.
The effect of the precise performance in the midterm tests is less clearcut. Considering
Chapter 5. 5.7. Conclusion Figure D2a, we see that we have a statistically significant relationship in the very low quantiles (below 20%) which suggests that good performance in the midterm tests is an indicator for better performance for very weak students who are expected to achieve a low number of points. However, the effect vanishes (in both a and b) for the 20 to 40% quantiles. For the median to 80% quantile students, good performance during the midterm also predicts a higher number of points in the final exam. For the very good students (upper quantiles), there is no significant relationship. Hence, irrespective of their midterm test outcome, these students are expected to achieve a high number of grade points in the final exam which is plausible.
How often a student made use of the MAD app does not seem to have an effect in the specification presented in FigureD3a, as suggested by the regression results before.
This holds across all quantiles. In contrast, Figure D3bmight imply a slightly negative relationship between participation in the MAD and exam performance for the very weak, i.e., the lowest 5% quantile.
Nevertheless, good performance in the MAD appears to be a good predictor for suc-cessful exam outcome, in particular for students who are expected to achieve only a low number of points in the exam. This holds for both specifications presented in FiguresD4a and D4b.
5.7 Conclusion
Our analysis evaluated several e-learning exercises during one semester accompanying the math-lecture designed for students of business and economics studies in their first semester. We found that positive learning gains are associated with students’ particip-ation in these exercises. The participparticip-ation in the midterm/practice tests accounts for a sizable increase in the points attained in the final exam. The number of submissions to the matrix app (MAD) was, however, insignificant. In addition to the participation effect, we find that students who had a higher performance in the e-learning
opportun-Chapter 5. 5.7. Conclusion ities (MAD as well as midterm/practice tests) were more likely to obtain higher scores in the final exam.
In our analysis we employ a rich set of control variables in order to approximate the causal effect of practice participation and performance on the exam points. We employ several robustness checks with regard to the selection of students as well as overfitting.
To counter the concern of overfitting, we employ variable selection techniques from the field of machine learning, namely Lasso, Random Forest, and xgBoost. In all cases, the different sets of selected variables did not change the measured effect of practice performance and participation significantly.
Finally, we used quantile regression of final exam points on the most important variables identified by the machine learning techniques to explore the question of how much the various performance groups in the final exam have benefited from practice and self-testing. We show that especially the students with lower points benefited the most from additional practice.
Therefore, our results suggest that giving students the possibility to self-test and practice the material in online settings with knowledge of correct response helps students to improve (math) exam grades.
Chapter 5. Appendix E
Appendix
D.1 Figures
Figure D1– Quantile regression coefficients: Number of taken midterm tests
0.2 0.4 0.6 0.8
0246810
Quantile of Attained Points in Exam
Coefficient
(a) Intersection
0.2 0.4 0.6 0.8
0246810
Quantile of Attained Points in Exam
Coefficient
(b) Union
Note: The panel shows the estimates for the coefficients of the number of midterm tests taken across quantiles. The quantile specific estimates are obtained from a quantile regression on the points obtained in the exam. Figure D1a presents the coefficients when the intersection of all variables selected by machine learning techniques are considered in Section5.6.1. FigureD1bpresents the coefficient when the union of all selected variables comprises the control set. The solid red horizontal line shows the value obtained in an equivalent OLS regression, while the dotted red lines indicate the 90%-confidence bounds of the OLS estimate. The shaded areas identify the 90% confidence bounds of the quantile regression estimates. Standard errors of the quantile regression coefficients have been calculated based on a wild bootstrap procedure with 5000 replications.
Chapter 5. Appendix E
Figure D2 – Quantile regression coefficients: Obtained points in midterm test
0.2 0.4 0.6 0.8
−0.10.00.10.20.30.40.5
Quantile of Attained Points in Exam
Coefficient
(a) Intersection
0.2 0.4 0.6 0.8
−0.2−0.10.00.10.20.30.4
Quantile of Attained Points in Exam
Coefficient
(b) Union
Note:The panel shows the estimates for the coefficients of the mean achieved points in the midterm tests across quantiles.
The quantile specific estimates are obtained from a quantile regression on the points obtained in the exam. FigureD2a presents the coefficients when the intersection of all variables selected by machine learning techniques are considered in Section5.6.1. FigureD2bpresents the coefficient when the union of all selected variables comprises the control set. The solid red horizontal line shows the value obtained in an equivalent OLS regression, while the dotted red lines indicate the 90% confidence bounds of the OLS estimate. The shaded areas identify the 90% confidence bounds of the quantile regression estimates. Standard errors of the quantile regression coefficients have been calculated based on a wild bootstrap procedure with 5000 replications.
Chapter 5. Appendix E
Figure D3 – Quantile regression coefficients: Number of submissions to MAD
0.2 0.4 0.6 0.8
−0.50.00.51.0
Quantile of Attained Points in Exam
Coefficient
(a) Intersection
0.2 0.4 0.6 0.8
−0.50.00.51.0
Quantile of Attained Points in Exam
Coefficient
(b) Union
Note:The panel shows the estimates for the coefficients of the number of submissions to the matrix app across quantiles.
The quantile specific estimates are obtained from a quantile regression on the points obtained in the exam. FigureD3a presents the coefficients when the intersection of all variables selected by machine learning techniques are considered in Section5.6.1. FigureD3bpresents the coefficient when the union of all selected variables comprises the control set. The solid red horizontal line shows the value obtained in an equivalent OLS regression, while the dotted red lines indicate the 90%-confidence bounds of the OLS estimate. The shaded areas identify the 90% confidence bounds of the quantile regression estimates. Standard errors of the quantile regression coefficients have been calculated based on a wild bootstrap procedure with 5000 replications.
Chapter 5. Appendix E
Figure D4 – Quantile regression coefficients: Achieved percentages MAD
0.2 0.4 0.6 0.8
−0.050.000.050.100.150.20
Quantile of Attained Points in Exam
Coefficient
(a) Intersection
0.2 0.4 0.6 0.8
−0.050.000.050.100.150.20
Quantile of Attained Points in Exam
Coefficient
(b) Union
Note:The panel shows the estimates for the coefficients of the mean percentage of correct answers for submitted solutions to the matrix app across quantiles. The quantile specific estimates are obtained from a quantile regression on the points obtained in the exam. FigureD4a presents the coefficients when the intersection of all variables selected by machine learning techniques are considered in Section5.6.1. FigureD4bpresents the coefficient when the union of all selected variables comprises the control set. The solid red horizontal line shows the value obtained in an equivalent OLS regression, while the dotted red lines indicate the 90%-confidence bounds of the OLS estimate. The shaded areas identify the 90%
confidence bounds of the quantile regression estimates. Standard errors of the quantile regression coefficients have been calculated based on a wild bootstrap procedure with 5000 replications.