6.6 Conclusion R4: Easy Adaptation to Multiple Types of Prediction: Through choosing the weighted
variants of the logistic and hinge loss functions, we adapted the system to the type or prediction necessary in this case.
Even though specialized models may outperform the application of the rather gen-eral matrix factorization framework done here, it is thus safe to conclude that the chine learning aspect of Detailed Feedback Machine Teaching can be provided by ma-trix factorization models, especially by the one introduced in Chapter 4 of this thesis.
The predictions generated are accurate and quickly available, even in the case where changed artifacts occur frequently.
7 Conclusions
7.1 Summary
Problem Statement By its very design, current Technology Enhanced learning sys-tems cannot convey knowledge that is not standardized, structured and most of all ex-ternalized. However, important knowledge such as intuition, experience or more gen-erally speaking implicit knowledge does not meet these requirements. Additionally, the structured externalization of knowledge in the form of electronic learning material is often impossible for economic reasons, as its creation is costly.
In traditional education, the apprenticeship with its master-student relationship is used to convey this knowledge. In this setting, the master explains to, discusses with and corrects the learner in a situation centered around the learned activity. This ap-proach is very effective. Yet, it by design requires the availability of a master in the field to the learner.
Approach MACHINETEACHING is proposed in this thesis to overcome these limita-tions: We notice that knowledge, even if not standardized, structured and externalized, can still be observed through its application. We refer to this observable knowledge as PRACTICED KNOWLEDGE. Machine Teaching is build upon this notion: Machine Learning techniques are used to extract machine models of Practiced Knowledge from observational data. The models thus capture traces of the Practiced Knowledge that went into the artifacts and processes, be it implicit or explicit. These models are then applied in the learner’s context for his support.
It is important to note that the people creating the artifacts or following the processes need not to be aware of their role in the system: They are not teachers, they are prac-titioners. Therefore, Machine Teaching focuses on thepractice, not theideal, just as the apprenticeship.
While both processes and artifacts can be the anchor in a Machine Teaching scenario, the automatic detection of processes still is an open field of research. As this is a pre-condition to a successful implementation of a Machine Teaching system, the thesis puts a focus on artifacts.
Levels of Feedback
We identified two important sub-classes of Machine Teaching: General and Detailed Feedback Machine Teaching.
General Feedback Machine Teaching General feedback in a way is like the grades given at school: Given an artifact, the Machine Teaching system determines its qual-ity. This is a straight-forward application of techniques from the field of supervised machine learning and the main effort in building a specific Machine Teaching System lies in deriving appropriate feature extraction procedures. Thus, General Feedback Ma-chine Teaching is readily applicable to a wide range of scenarios where the feature engineering task is either straight-forward or has been studied to great extend. This in-cludes tasks related to text, images and of course artifacts which exhibit feature-vector
7.1 Summary structure themselves. The application presented in this thesis, learning to write for a specific community, shows that the approach is viable and that the predicted quality of the artifact can in fact be very close to the one perceived by the community.
Detailed Feedback Detailed Feedback Machine Teaching aims at providing feedback regarding the attributes of the artifact to the learner. To do so, the machine learning model needs to have a notion of these artifact and their relations. This, in principle, provokes the creation of a specific machine learning model for each detailed feedback application of Machine Teaching. However, it has been shown in this thesis that De-tailed Feedback Machine Teaching often exhibits a striking similarity to the task of a recommender system. Where the latter suggests items to customers, the former “sug-gests” attributes to artifacts. Despite this obvious similarity, Machine Teaching poses additional requirements onto the underlying algorithm not usually occurring in Rec-ommender Systems. These include a greater variety in the attributes of artifacts, the frequent prediction upon yet unseen artifacts and the real-time nature of a Machine Teaching system. Last but not least, Detailed Feedback Machine Teaching imposes con-sistency requirements for one artifact as a whole, while a Recommender System is usu-ally focused on predicting the rating a user would associate with a single item.
Matrix Factorization Model and Method This thesis presented a novel model and al-gorithm based on factor models that substantially generalizes the state-of-the-art in this field to address these requirements uniquely found in Machine Teaching. The al-gorithm is a valuable contribution to the field of Recommender Systems in its own right as it is the first to be able to predictrankingsas opposed toratings. Additionally, it forms a hybrid Recommender System that can use known features of the user (artifacts) and items (structural elements) in addition to the collaborative effect present in such data.
The base algorithm as well as the extensions thereof proposed in this thesis have shown favorable performance on standard Recommender Systems evaluation data sets avail-able. It is important to note that the algorithm excels especially in what is called the new-user problem in Recommender Systems: Given a new, yet unseen user with some rated items and a trained model based on other users, the goal is to predict well for this new user. While this situation is theexceptionin Recommender Systems, it is thenorm in Machine Teaching: every time the learner presents the system with an artifact, that artifact constitutes a “new user”. Thus, excellent predictive performance on this task is crucial for Machine Teaching.
Machine Teaching for Software Engineering The domain of learning to program well within a software framework is modeled as a task for Detailed Feedback Machine Teaching by forming a caller-callee matrix of representative code. This matrix stores every call to the software framework from the code analyzed, where each row repre-sents one method or class of the analyzed code and each column reprerepre-sents a possibly callable method of the framework. The goal of a Machine Teaching system when con-fronted with a partial row is to predict missing calls. As we have shown in the empirical
evaluation, the model is capable of capturing the structure in this matrix and can there-fore compute meaningful feedback for the learning programmer.
This application exemplifies the expressive power of the factor model presented in this thesis, as the model was applied to the problem in a straight-forward manner. So even while purpose-built models are likely to perform even better, the factor model can serve as the underlying model for a wide range of Machine Teaching applications.
It therefore provides Detailed Feedback Machine Teaching with a foundation just as supervised machine learning forms the base of General Feedback Machine Teaching.