SPEC is a unified spectral feature selection framework for supervised, un-supervised, and semi-supervised feature selection. It is based on three univari-ate formulations for feature evaluation and is of a filter model. We show that families of effective algorithms can be derived from the framework. We con-duct robustness analysis based on perturbation theory. The analysis enables us to obtain better understanding of the behavior of the SPEC in a noisy learning environment.
The proposed framework consists of three components: the similarity ma-trix S, the ranking function ˆϕ(·), and the spectral function γ(·). A proper configuration of the framework ensures good performance. Based on our exper-imental results and observations, we offer the following guidelines for configur-ing SPEC: (1) The similarity matrix depicts the relationship among samples.
A matrix which reflects the true relationships among samples is important for SPEC to select good features. (2) In noisy learning environments, either ˆϕ3(·) or a high-order rational function γ(·) is helpful for removing noise. (3) For data with a clear spectrum gap, using ˆϕ3(·) may be very effective. Otherwise
ˆ
ϕ2(·) could be more promising. Compared with ˆϕ3(·), ˆϕ2(·) is less aggressive and usually provides robust performance. (4) SPEC generates feature weight-ing algorithms. However, most feature weightweight-ing algorithms do not consider feature redundancy, which may hurt learning performance [212]. To address the problem, we will propose a multivariate formulation for spectral feature selection in Chapter 3.
Chapter 3
Multivariate Formulations for Spectral Feature Selection
Redundant features are those that are relevant to the target concept, but their removal has no negative effect. Usually, a feature becomes redundant when it can be expressed by other features. Redundant features unnecessarily increase data dimensionality [89], which worsens the learning performance.
It has been empirically shown that removing redundant features can result in significant performance improvement [69, 40, 56, 210, 6, 43]. In the last section, we introduced the SPEC framework for spectral feature selection. We notice that the feature evaluation criteria in SPEC are univariate: features are evaluated individually, therefore the framework is not capable of handling redundant features.
Example 13 An example of a redundant feature
Assume we have three featuresF1, F2, andF3. Among the three features, F3 can be expressed as a linear combination ofF1andF2:
F3=aF1+bF2, a, b∈R.
Given any function containing the three features, we can write it as a function that contains only F1 andF2:
φ(F1, F2, F3) =φ(F1, F2, aF1+bF2) =φ0(F1, F2).
Therefore, in this case,F3is redundant due to the existence ofF1andF2.
Spectral feature selection can handle redundant features by evaluating the utility of a set of features jointly. In this chapter, we study two multivariate formulations for spectral feature selection, one based on multi-output regres-sion [72] with an L2,1-norm regularization, and the other based on matrix comparison. We analyze their capabilities for detecting redundant features, and study their efficiency for problem solving. Before we present the two for-mulations, we first study an interesting characteristic of the SPEC framework, which we introduced in the last chapter. We show that SPEC selects features 55
by evaluating their capability of preserving the sample similarity specified by the given similarity matrixS. Based on this insight, we present two multivari-ate formulations for spectral feature selection.
3.1 The Similarity Preserving Nature of SPEC
As shown in Chapter 2, given a similarity matrixS, SPEC selects features aligning well with the top eigenvectors ofL. HereL is the normalized Lapla-cian matrix derived fromS. This fact brings us to the conjecture that if we construct a new sample similarity matrix K, using the features selected by SPEC,Kshould be similar toS, in the sense that if the two samples are sim-ilar according toS, they should also be similar according toK. To precisely study the similarity preserving nature of SPEC, we reformulate the relevance evaluation criteria used in the SPEC framework in a more general form:
maxFsub Basically, we want to find a set of selected features, Fsub, such that the objective specified in Equation (3.1) can be maximized. In the above equation, ˆfand ˆSare the normalized feature vector and the normalized sample similarity matrix derived fromf andS, respectively. It is shown in [155] that solving the following problem
maxK0Trace (KS) st. Trace (K)≤1, (3.2) will result in a kernel matrixK, which preserves the sample similarity specified in S. Here, the constraint K0 requires the matrixK to be positive semi-definite. We can write Equation (3.1) in the form of Equation (3.2),
maxFsub con-taining only the features inFsub. Thus, we have the following equation:
maxFsub
This equation shows that max
Fsub
P
F∈Fsub ˆf>Sˆ ˆf will select a set of features
Fsub, such that the linear kernel constructed from XFsub can preserve the pairwise sample similarity specified in ˆS. In other words, we can say that the features in Fsub have a strong capability of preserving the pairwise sample similarity specified in ˆS. We can also show this in a more intuitive way: since ˆf>Sˆ ˆf =P
i
P
isˆijfˆifˆj, assuming that features are normalized (kˆf k= 1), to obtain a large value from Equation (3.1), a feature must assign similar values to the samples that are similar according to ˆS. This ensures that the feature has the strong capability of preserving the sample similarity specified in ˆS.
Example 14 Measuring consistency between matrices
Trace (KS) can be used to measure the consistency between matrices. To show this, we generated a two-dimensional data set with three classes, whose distribution is shown in Figure 3.1(a). We then generate noise-contaminated data sets by adding different levels of noise to the data set.
Figure 3.1(b), (c), and (d) correspond to the data sets containing 30%, 60%, and 90% of noise, respectively. We construct linear kernels on both the origi-nal data set and the noise-contaminated data sets, and compute Trace (KS) to measure the consistency between matrices. Here Sis the linear kernel constructed on the original data set, andKis the linear kernel constructed on either the original data set or a noise-contaminated data set. From the figures the similarity relationships among samples are perturbed propor-tionally to the level of the noise added to the data. And correspondingly, the value of Trace (KS) decreases. WhenK=S, Trace (KS) = 1.282, while whenKis constructed on the data set containing 90% of noise, Trace (KS) decreases to 0.7994.
In the following, we show how SPEC can be reformulated in the form of Equation (3.1). We first study a simple case, in which the spectral matrix function,γ(·), is not applied. Using the following theorem, we show that with different definitions of ˆf and ˆS, the three feature ranking functions ϕ1(·), ϕ2(·), and ϕ3(·) can be written in a common form: max
Fsub
P
F∈Fsub ˆf> Sˆ ˆf.
Here, ˆf and ˆSare the normalized versions off andS.
Theorem 3.1.1 LetSbe a similarity matrix, andDandLbe its degree and normalized Laplacian matrices, respectively. SPEC selects k features, which maximize the following objective function:
arg maxFi1,...,Fik
Xk j=1
ˆfi>j Sˆ ˆfij. (3.4)
Trace = 1.1614
−2 0 2 4 6 8
−2
−1 0 1 2 3 4 5 6 7
Trace = 1.1056
−1 0 1 2 3 4 5 6
−1 0 1 2 3 4 5 6
Trace = 0.9915
−1 0 1 2 3 4 5
−0.5 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5
Trace = 0.8707
−0.50 0 0.5 1 1.5 2 2.5 3 3.5
0.5 1 1.5 2 2.5 3 3.5 4
FIGURE 3.1: Measure consistency between matrices via Trace (KS).
Whenϕ1(·)is applied,ˆf andSˆ are defined as SPEC, features are evaluated independently; therefore, usingϕ1(·) to selectk features can be achieved by picking the topkfeatures that have the smallest ϕ1(·) values. This process can be formulated as the following optimization problem:
Note that in the above equation, features are evaluated independently. Addi-tionally, Here we assume the features have been normalized, therefore all have unit norm.
In the case ofϕ2(·), it is easy to verify thatkˆfk= 1. Since the first eigen-value of the normalized Laplacian matrixLis always zero, we haveξ1>Lˆx= 0 for anyx∈Rn×1. Based on these two facts, we can verify that Equation (3.6) holds.
Similarly, in the case ofϕ3(·), the following equation holds:
f>Uk(2I−Σk) U>kf = Xk j=2
(2−λj)α2j =ϕ3(F).
This proves the equivalence whenϕ3(·) is used.
Theorem 3.1.1 shows that when ϕ1(·) or ϕ2(·) is used, SPEC tries to preserve the sample similarity specified byD−12SD−12, which is the normalized sample similarity matrix. When ϕ3(·) is used, SPEC tries to preserve the sample similarity specified byUk(2I−Σk) U>k, which is derived fromL by adjusting the leading eigenvalues and discarding the tail eigenpairs. Inϕ1(·) andϕ3(·), the features are first reweighted byD12f, which forms the density reweighted features [74]. And they are then normalized to have the unit norm.
This step emphasizes the elements in a feature vector, which correspond to the samples from a neighborhood with dense sample distribution. Inϕ2(·), there is an additional orthogonalization step: features are made to be orthogonal to ξ1, and then normalized to have unit norm. This step removes ξ1 from consideration. As we mentioned, by aligning closely with ξ1, a feature can achieve a largeϕ1(·) value. However,ξ1only captures the density information of the data. The orthogonalization step ensures that we will not assign high relevance scores to features that align closely withξ1.
Similarly, when the spectral matrix functionγ(·) is used in SPEC, using the following theorem, we show that the three feature ranking functions ˆϕ1(·),
ˆ
ϕ2(·), and ˆϕ3(·) can also be formulated into the form: max
Fsub
P
F∈Fsub ˆf> Sˆ ˆf.
Theorem 3.1.2 LetSbe a similarity matrix, andDandLbe its degree and normalized Laplacian matrices, respectively. Also letγ(·)be a spectral matrix function. SPEC selectsk features that maximize the objective function:
arg maxFi
Theorem 3.1.2 can be proved in a way similar to Theorem 3.1.1. There-fore we omit its proof. Theorems 3.1.1 and 3.1.2 together demonstrate the similarity preserving nature of SPEC.
In SPEC, features are evaluated independently. A direct consequence of this is that redundant features cannot be properly handled by the SPEC framework. Redundant features unnecessarily increase dimensionality and can worsen learning performance. They need to be removed in the feature selection process. To achieve this, we propose in the following sections two multivari-ate formulations for spectral feature selection, which are able to evalumultivari-ate the utility of a set of features jointly. We show that the multivariate formulations for spectral feature selection can identify redundant features effectively.