2.3 Analysis procedures and statistical tests
2.3.4 Classification of connection existence
To assess how well an explanatory framework accounted for patterns in the existence of cortico-cortical connections, we used a range of classification procedures. By
evaluating how well the structural measures enabled the classification of connection existence, individually or in conjunction, we were able to generate insights into the principles underlying the characteristics of connection existence.
Linear discriminant analysis
To assess the distribution of present and absent projections in the cat cortex across the variables |∆type| and ∆dist more closely, we performed a linear discriminant analysis (Klecka, 1980; Burns and Burns, 2008). A linear discriminant analysis determines a linear combination of predictive variables that optimally separates distinct classes of a dependent variable. We used |∆type| and ∆dist as predictive variables, and existence of projections as the dependent variable. Given the non-significant correlation of relative projection frequencies with |∆level| (see Section 3.2.1), we did not include |∆level| into the linear discriminant analysis. We assumed uniform prior probabilities for the two classes of the dependent variable (‘absent’ and
‘present’). The linear discriminant analysis then provides a posterior probability for each combination of |∆type| and∆dist, which can be used to classify new data points (unknown connections) as either absent or present.
To account for the fact that not all combinations of the predictive variables can occur equally often (e.g., combinations of |∆type| = 1 and ∆dist= 1 are frequent in this cortical parcellation, while combinations of |∆type| = 4 and ∆dist= 4 are not), we normalised the numbers of absent and present projections of a specific combination of |∆type| and ∆dist by the maximally possible number of co-occurrences of that combination. This resulted in proportions %absentand %present of projections at each point in the predictive variable space. Note that %absent + %present , 100, which reflects the fact that there is a remaining percentage of projections which have not been examined. To transform the resulting percentages into cases suitable as input for the linear discriminant analysis, we constructed, for each combination of |∆type| and ∆dist,Na= %absent cases with the respective values of the predictive variables and a dependent variable rating of ‘0’ (absent), and Np= %present cases with the same predictive variables but a dependent variable rating of ‘1’ (present). Compared to using the raw data as input for the linear discriminant analysis, this procedure adjusts the relative importance of examined projections by taking into account how thoroughly the underlying predictive variable space was sampled.
Cross-validation was performed by randomly excluding 10% of the data from the training set and using this test set to validate the obtained model. We tested model
2.3. Analysis procedures and statistical tests
performance at seven different classification thresholds, starting at 0.60 and increas-ing in 0.05 increments to 0.90. Connections were assigned the status ‘present’, if the posterior probability for the presence of connections at their associated |∆type| and
∆distwas equal to or larger than the classification threshold, and assigned the status
‘absent’, if their associated posterior probability was equal to or smaller than 1 minus the classification threshold (i.e., 0.40, decreasing in 0.05 increments to 0.10). We did not classify the status of connections with associated posterior probabilities that fell into the intermediate range. We computed prediction accuracy to assess classification performance, where accuracy equalled the number of correct predictions divided by the total number of predictions. We calculated this measure separately for predictions assigning either the status absent (correct absent), the status present (correct present) or all predictions (correct total). We performed 200 cross-validation cycles and report averaged results.
Support vector machine
In our analyses of the relative merit of different explanatory frameworks in the macaque cortex (Section 3.3), we combined the structural measures in a different probabilistic predictive model for classifying the existence of projections. We built this model using a binary support vector machine classifier (i.e., used for two-class learning), which received the structural measures associated with the projections as independent variables (features) and information about projection existence (i.e., projection status ‘absent’ or ‘present’) as the dependent variable (labels, comprising two classes). Euclidean distance, absolute log-ratio of neuron density and absolute log-ratio of cortical thickness were used as features in different combinations.
For training the support vector machine classifier, we used a linear kernel function and standardised the independent variables prior to classification. Moreover, we assumed uniform prior probabilities for the learned classes and assigned a symmetric cost function, that is, all types of errors were weighted equally. Classification scores obtained from the trained classifier were transformed to the posterior probability that an observation was classified as ‘present’, ppresent. To assess performance of the classification procedure, we used five-fold cross-validation. To this end, we randomly partitioned all available observations into five folds of equal size. After training the support vector machine classifier on a training set comprising four folds, we used the resulting posterior probabilities to predict the status of the remaining fold (20% of available observations) that comprised the test set. Similarly to the procedure followed in the linear discriminant analysis, we used two classification
rules derived from a common threshold probability. (1) We assigned the status
‘present’ to all observations whose posterior probability exceeded the threshold probability, that is, observations with ppresent>pthreshold. (2) We assigned the status
‘absent’ to all observations with ppresent<1−pthreshold. The approach was applied to thresholds from pthreshold= 0.50 to pthreshold= 1.00, in increments of 0.025. By increasing the threshold probability, we therefore narrowed the windows in the feature space for which classification was possible. For thresholds of pthreshold<= 0.50, the classification windows would overlap. In particular, there would be an overlap between parts of the feature space corresponding to classification as ‘present’ with parts corresponding to classification as ‘absent’, and observations would therefore be classified twice. For this reason, we did not consider thresholds below pthreshold= 0.50.
For each threshold, we computed performance as described below and averaged results across the five cross-validation folds. To make performance assessment robust against variability in the partitioning of observations, we report performance measures averaged across 100 rounds of the five-fold cross-validation.
We assessed classification performance by computing prediction accuracy, the fraction of correct predictions relative to all predictions. Accuracy was also separately assessed for positive and negative predictions, yielding precision and negative predictive value as the fraction of correct positive or correct negative predictions relative to all positive or negative predictions, respectively. We also computed which fraction of observations in the test set was assigned a prediction at a given threshold. As further performance measures, we computed sensitivity (true positive rate) and specificity (true negative rate) at the evaluated thresholds. We also computed the false positive rate (1−specificity). To quantify performance based on sensitivity and specificity, we computed the Youden indexJasJ= sensitivity + specificity−1 (Youden, 1950; Fluss et al., 2005). J is a measure of how well a binary classifier operates above chance level, with J= 0 indicating chance performance and J= 1 indicating perfect classification. Since J is defined at each threshold, to obtain a single summary measure we computed the mean ofJ across the more conservative thresholds ppresent= 0.85 to ppresent= 1.00 for all 100 cross-validation runs. Results did not change if the maximum J across all thresholds was considered instead (Supplementary Figure C.2B).
To assess statistical null performance of the classification procedure, we performed a permutation analysis. The analysis was equal to the classification procedure described above, with the exception of an additional step prior to the partitioning of observations into cross-validation folds. Here, for each round of cross-validation, the labels were randomly permuted. Thereby, the correspondence between features and true labels of observations was removed. In the permutation analysis, we used
2.3. Analysis procedures and statistical tests
Euclidean distance and the absolute log-ratio of neuron density as features, based on the feature combination that led to the best results, and averaged performance measures across 1000 rounds of five-fold cross-validation.
Logistic regression
In our assessment of the relative predictive power of microscopic and macroscopic structural measures (Section 3.4), we performed multivariate logistic regression analyses using projection existence as the binary dependent variable and differ-ent combinations of the relative structural measures as covariates. Specifically, we considered |log-ratiodensity|, geodesic distance, |∆soma cross section|, |∆spine count|,
|∆spine density|, and |∆tree size|. The relative structural measures were converted to z-scores, so that the resulting regression coefficients were standardised. We also included a constant intercept term in each model. All covariates were entered into the model simultaneously. We report the standardised regression coefficients, the t-statistic, and its associated p-value.
For the logistic regression, we assessed model classification performance in three different ways. First, we calculated the generalised coefficient of determination, R², adjusted for the number of covariates, which indicates which proportion of the vari-ance in the dependent variable is explained by the covariates. Second, we computed the Youden indexJ (Youden, 1950; Fluss et al., 2005), whereJ= sensitivity + speci-ficity−1. As mentioned above, by taking into account both sensitivity (true positive rate) and specificity (true negative rate), the Youden index is a comprehensive sum-mary measure of classification performance. We considered values ofJbelow 0.25 to indicate negligible classification performance, values of 0.25 and above to indicate weak performance, values of 0.40 and above to indicate moderate performance, and values of 0.50 and above to indicate good classification performance. Third, we calculated classification accuracy, that is, which proportion of all predictions was correct.
Parts of this section have been published in Beul et al. (2015), Beul et al. (2017) and Beul and Hilgetag (2019a).