Research Questions
It also follows that the appropriateness of a feature is task-dependent: A feature that is very successful in systems for one task does not necessarily perform equally well in systems built for other tasks.
As General Feedback Machine Teaching is a new task from the perspective of super-vised machine learning, the main research question can be phrased as:
Can features be found that allow a supervised machine learning system to be part of a General Feedback Machine Teaching system?
This question is addressed via an example in Chapter 3. In that Chapter, features are proposed and evaluated for the task of estimating the rating of web forum posts. It will be shown that features in fact can be found for this task.
To conclude, General Feedback Machine Teaching systems can be built and thus it is prudent to discuss the more ambitious goal of providing the learner with suggestions regarding her artifact through a Detailed Feedback Machine Teaching system.
2.5 Detailed Feedback Machine Teaching
Again, we start the description of Detailed Feedback Machine Teaching with its defini-tion, closely following that of Machine Teaching in general:
Definition 8(Machine Teaching). ADETAILEDFEEDBACKMACHINETEACHINGsystem supports learners in the creation of artifacts by providing suggestions regarding these artifacts.
To provide these suggestions, a Detailed Feedback Machine Teaching system needs a ma-chine model of the artifact in question which captures structure of the kind of artifact in ques-tion. These models are extracted from similar artifacts created by more experienced people by means of machine learning.
Based upon these models and the artifact presented to the system by the learner, the Machine Teaching system generates suggestions and presents them to this learner.
The remainder of this section discusses the possible choices regarding a machine learning model for this task. We will thereby identify open research questions that are summarized at the end of this section.
Machine Learning Models
To provide suggestions regarding an artifact, a Detailed Feedback Machine Teaching system needs a grasp of the artifact’s structure. Similarly to the artifact features intro-duced above, this structure of the artifact is represented by structural elements of the artifact, which we define as:
Definition 9(Structural Element). ASTRUCTURAL ELEMENTof an artifact is a feature of the artifact that is mutable by the user, respectively learner of the system.
Structural elements are typed just as features are. Common types include binary, real valued and one-out-of-n.
One artifact may be represented by a vector x of these structural elements, where each dimension corresponds to one structural element. The value of the vector of one artifact in that dimension is the value of the structural element. As an artifact is not required to have all structural elements, the vector may be sparse.
The goal of Detailed Feedback Machine Teaching can then be stated as: Given the structural elements of an artifact, suggest changes to the values of these or additional structural elements. Thus, the main difference to the goal of General Feedback Machine Teaching from a machine learning perspective is that instead of predicting a labelyfor a given samplex, the system is now asked to predict changes to the samplex.
In contrast to General Feedback Machine Teaching, this task does not map directly to one specific machine learning technique. In fact, one could design a statistical model for each artifact type and fit that model to the available data in principle. Graphical models as introduced e. g. in [Bis06] (Chapter 8) provide the conceptual framework to build these models and henceforth machine learning algorithms for them.
The Connection to Recommender Systems
While the task of designing a model for each artifact type seems daunting, the following observation allows us to provide a machine learning model in a large set of Detailed Feedback Machine Teaching settings:
Observation: The process of suggesting (values of) structural elements of artifacts is strikingly similar to that of suggesting products or services to users in a Recommender System.
Recommender Systems are the enabling technology for many e-commerce vendors and have found great research interest. The term “Recommender System” is often only implicitly defined. For the sake of precision, the term shall be defined as follows for the remainder of this thesis:
Definition 10 (Recommender System). A RECOMMENDER SYSTEM suggests items to users. The goal is to suggest those items to a user she is likely to like.
The recommendations are based upon available data about the user and the items, including interaction data between the users and items, e. g. in the form of user-supplied ratings.
In most cases, a recommendation is computed by first estimating the preference of the user for the items and then presenting those with the highest estimates to the user. This core predic-tion logic of the system shall be referred to asRECOMMENDERALGORITHM.
Following this definition, the observation above allows us to define Recommender based Machine teaching as:
2.5 Detailed Feedback Machine Teaching Definition 11 (Recommender based Machine Teaching System). A RECOMMENDER BASED MACHINE TEACHING SYSTEM suggests values of structural elements for artifacts.
The goal is to suggest those values that fit the artifact well.
The suggestions are based upon available data about the structural elements and the arti-facts, including in particular similar artifacts as represented by their structural elements.
Requirements of a Recommender System
Despite this striking similarity, a Recommender based Machine Teaching system poses a specific set of requirements on the underlying recommender algorithm:
R1: Accurate solutions for partial artifacts: In its application, the Machine Teaching system needs to suggest values for structure elements based on partial artifacts as it is to support the learner during the creation of the artifact. Every time the learner changes the artifact in ways that alter the structural elements, the system needs to compute new or adapted recommendations.
This is similar to the “new user” problem discussed in the Recommender Systems literature: A new user who has rated just a few items expects meaningful sugges-tions from the recommender system. The presence of new artifacts in this sense is the norm in Machine Teaching scenarios, while it is the exception in a eCommerce recommender system.
R2: Interactive Performance: The Machine Teaching System shall be deployed such that it can provide accurate suggestions just-in-time: The Machine Teaching sys-tem must keep up with the speed the learner manipulates the artifact or, at least, the rate at which she demands suggestions from the system.
This is often less of a concern for commercial recommender systems, as the rate at which people buy items is considerably lower than the rate at which learners can manipulate artifacts.
R3: Recommendation Coherence: The recommendations given by the Machine Teach-ing System may be subject to coherence restrictions such as in the followTeach-ing ex-ample:
Example 4. In the programming example in Section 2.2.2, the system should only recommend an operation on aStringobject if an object of that type is either present or its creation is also suggested.
Note that coherence requirements may be formalized in some cases, while in the more general case they are to be inferred from the data.
R4: Easy Adaptation to Multiple types of prediction: The structure elements of arti-facts may be of various types. Thus, the Machine Teaching System needs to be easily adaptable to these changing types. Recommender algorithms, in contrast, are most often developed solely to predict the numerical rating a user would at-tach to a given item.
Available Recommender Algorithms
The Recommender Systems literature, e. g. in the survey paper [AT05] distinguishes two different approaches: Memory based Recommender Systems and Model based Recommender Systems.
Memory based Recommender Systems:1 To compute the predicted ratings, these sys-tems search for the users which are most similar to the one the prediction is made for. The notion of similarity can be based on features as defined above, e. g. their age or address, or it is based on past purchase data. Given those similar users, the predictions is computed as the, potentially weighted, average of the labels given to the item in question by those users.
The same approach can be applied from an item perspective: A list of items is retrieved that are similar to the one in question. The weighted average over the labels “given” to the user in question is the prediction of the system.
Model based Recommender Systems: Given past data, these systems build a model of the function that associates a user-item pair with a label. One approach to do so is to derive a function from the features of the users and items to the label. As for Memory based Recommender Systems, the model may also be defined on the past purchase data of this and other users. Systems in that category are referred to asCollaborative Filtering Systems. Systems that make use of both user and item features and the collaborative information are referred to asHybrid Recommender Systems.
We will now investigate the suitability of these classes of Recommender Systems in the light of the requirements described above.
Memory based recommender systems need to perform a search for similar artifacts whenever the learner requests assistance. In the desirable case where the Machine Teaching system can draw upon a large database of artifacts, this search considerably limits the real time applicability of the system, violating requirement R2. We therefore exclude memory based recommender systems from further analysis.
As stated in R1, a Machine Teaching System is frequently faced with new artifacts.
These artifacts have only very few values for their structural elements and therefore provide little input to the recommender system to build its recommendation upon. It is thus desirable to use a hybrid recommender system which is capable of using addi-tional features of the artifact and the structural element to enhance its prediction accu-racy.
Remaining Research Questions
However, to the best of our knowledge there is no model based hybrid recommender system that meets all of the requirements above. In Chapter 4, we present a novel recommender algorithm that meets these requirements. The application of this new algorithm to a Detailed Feedback Machine Teaching System for Software Engineers is subsequently presented in Chapter 6.