Data Mining

Many techniques used for MSR origin from the field of data mining. Data mining can be de-scribed as the “extraction of implicit, previously unknown, and potential useful information from data“ [32]. Here, raw data constitute recorded facts, whereas information means all underlying observations that can describe the data, e.g., by patterns. In contrast to the data, the information is not visible at first sight. A lot of approaches exist to uncover valuable information from data stored in databases. Data mining techniques can be of descriptive or predictive nature. Descriptive techniques reveal facts that are already there, but the infor-mation has to be filtered or put into a comprehensible format. In predictive data mining, the goal is to use some information that is already there to forecast characteristics of other information, e.g., prediction of future trends. Besides software engineering, data mining has a wealth of applications, e.g., in marketing, health, and bio informatics.

The main goal of data mining is to find patterns that fit the data and as such make the data understandable and interpretable. For finding nontrivial patterns in data, a lot of automated processes exist. Generally, desired patterns have to reveal meaningful characteristics of the data for a better understanding (descriptive) or are able to make powerful predictions about a specific outcome (predictive) [33]. Patterns always capture the underlying structure of the data. Such a pattern of software evolution can for example be a rule like:If file A and B are changed, often file C is changed as well. In data mining research, a lot of techniques that origin from the field of machine learning are used.

Generally, a selected element from the input data is referred to as aninstance. Often, an instance is represented by a set of attributes, which are defined asfeatures(e.g., [34]).

Features usually are multi-dimensional and, thus, represented as vectors.

The output of machine learning algorithms can be of qualitative nature or of quantitative nature. Quantitative output measures give information about the relationships of the data. In the case of qualitatively measured output, we get a finite set of categorical or discrete values that put the data into categories [35]. Here, the input data is used to predict the output. This type of learning task is calledsupervised learning. This means, that the learning process takes place with knowing the outcome for a set of instances. The input data, calledtraining set, is used to create apredictor. The predictor is then able to classify so far unseen data.

For assessing the goodness of a predictor, it usually gets applied to a set of new, unseen data, thetesting set. For the testing set, the expected output of the predictor is known and compared with the predictors actual output.

In Figure 2.3, supervised learning is illustrated on the right side. There we have a set of labeled (green,red) instances on the top serving as training data. Once the predictor is learned, all new instances can be labeled as green or red as well. This way, the data is separated according to the labeling. In contrast, unsupervised techniques deal with input

Foundations 18

Unsupervised Supervised

Figure 2.3.: Unsupervised vs. Supervised Learning.

data where no information about the outcome is available. Following the example in Fig-ure 2.3, the unlabeled data points on the top left are the input. There, no predictor can be learned, but structural properties of the data. Thus, the output could look like the grouped instances on the bottom of the picture. The labels in software evolution often indicate if a bug is within a revision which can simply be described with 1 for true and 0 for false.

Besides numerical values, categorical values, e.g., for the severity of bugs, are possible. In these cases, supervised learning is possible. Unfortunately, often the data for other mining tasks has no such labels available, e.g., for the roles of software developers. In this scenario, either unsupervised methods are available or the usage of heuristics, the distribution of the data, or expert knowledge to classify a part of the data can be taken into account to get such a label and make the data suitable for other learning tasks.

Learning Techniques

The following techniques fall into the category of supervised learning. Here, two main tasks of prediction exist:regression– where the output is measured quantitatively and clas-sification– where the output is of qualitative nature. Both can be seen as an approach to approximate some kind of function [35]. We describe the ideas behind the relevant tech-niques for this thesis in the following.

19 2.2. Mining Software Repositories 1. Linear and Polynomial Regression: The application of regression methods is the modeling of relationships between input and output variables. For a numeric out-put and numeric variables polynomial regression can be used to model the com-bination of the variables. Generally, this relationship can be expressed by f(x) = a₀+a₁x+a₂x²+...+anxⁿwitha₀, ...,anbeing the variables to model. In the linear case the degree nequals 1. Higher order regression is called polynomial regression according to the type of the resulting curve. Different algorithms exist to find the best fit, e.g., least squares. In software evolution, often curves of trends such as the growth of the system are calculated to describe the course.

2. k-Nearest Neighbor: In practice, nearest neighbor methods are used to classify unseen data by available labeled data that is similar. There each unseen instance is compared to other already known instances using a distance measure. In doing so, the class of the closest instance is assigned to the new one. Fork-Nearest Neighbor thekclosest entities are considered and a majority vote on the classes determines the class of the unseen instance. Unusually, the Euclidean distance is used for the computation of the distance.

3. Decision Trees and Random Forests: Another way to learn a predictor for classifi-cation as well as for regression is to use decision trees. They are used to classify instances by conditions based on the feature set. Here, a tree structure is learned where the interior nodes represent the decisions with two or more branches and the leaf nodes imply the outcome, i.e., the predicted class or probability. On the whole, for each combination of input variables, a path to a decision exist. Although a bunch of algorithms exist for learning, most of them adhere to the following main idea (ID3/C4.5) [34]: The algorithm builds the tree from the root to the leaves starting with the determination of the most powerful node, if taken alone, as the root node.

Then, for all reasonable values a successive node is created and, again, it is tested which attribute is the best to test in this place. Altogether, the algorithm implements a greedy search without backtracking [34]. One big advantage of decision trees is that they are easy to interpret if they are not too large.

Closely related, for random forests a set of decision trees is build based on a random-ized algorithm. The randomness is generated by searching for the most prominent feature in a randomly generated feature subset. Then, the nodes are divided according to the evaluation on this random subset. As a result, a combination of the most consis-tent predictions is used. In comparison with basic decision trees, random forests are often more accurate, but decision trees may be more valuable in terms of knowledge representation and interpretation.

4. Threshold Learning: In mining software repositories, often the attributes under inves-tigation are measured by software metrics, e.g., the lines of code to some point in time or the number of developers involved. Usually, more than one metric is of interest

Foundations 20 and, thus, the common trend of the project is harder to interpret. To deal with such metric sets, it can be of support to classify the metric values into problematic or not based on proper boundaries, i.e., thresholds. To calculate such thresholds, it is com-mon to use a PAC (Probably Approximately Correct) learner [36]. In the algorithm, ad-dimensional axis-aligned rectangle is learned which splits the data according to their label (inside and outside of the rectangle) withdthe number of attributes.

For advanced applications, especially when using machine learning, the data has to be prepared carefully. If the approaches work in the given context, strongly depends on the data. We now review different possibilities to transform data into a suitable format. In many machine learning algorithms, the selection of important features is integrated. Nev-ertheless, often they reach a better performance when applying a selection beforehand [32].

Such techniques include ranking, filtering, and wrapping. Another widespread method is sampling. There, subsets of the data are created based on the distribution of the data. This is especially applicable in larger scale applications. A technique which is relevant for our work, is the transformation of multi-class problems into a two-class problem. For this, it is common to split the data into two-class problems (called one-vs.-rest), but also pairwise classification can aid to solve this problem [32]. In this scenario, a classifier is learned for every couple of classes and a majority vote decides about the final classification.