7.6 Concordance with manually selected points of interest
7.6.1 Data
Finally, with regard to the applications in neurobiology (Chapter 9), it is of interest to analyse how algorithmic column selection from imaging movie matrices compares to manual column (pixel) selection by a domain expert, which is a common data analysis method for imaging movies (Section 4.3.1).
Clearly, manual pixel selection does not involve explicit optimisation of a norm reconstruction criterion. However, the glomerular signals contribute a lot to the norm of the movie matrix (Section 4.2). As the test data from this section has been used for a biological study on glomerular signals [59], we can assume that the domain expert has focussed on selecting signals that could be interpreted as being the pure (non-mixed) glomerular signals.
The evaluation was performed on calcium imaging movies from the honey-bee AL (Fura-2-dextran staining, Section 4.1). The dataset is described and analysed later in the section on odour concentration coding (Section 9.3) and further details can be found in [59, 189].
Manual ”pixel of interest” selection (Section 4.3.1) was performed by a domain expert (PhD student in neurobiology/neuroimaging). The selection process was guided by visualisations of the correlation between neighbouring pixels (”correlation image”, Section 4.3.1) and by visualisations of odour response patterns using false-colour coded images [Appendix][59]. Pixels were selected by mouse clicks in a GUI and pixel coordinates were logged to a file.
The column selection algorithms were applied to movies that had been preprocessed as described in Section 9.1.4, however without movement cor-rection so as to avoid image cropping and systematic coordinate shifts (with respect to the manual solution) by image alignment. Instead, a subset of 12 movies with minimal or no animal movement was chosen for the evaluation.
Correlation images and coordinates of the manually chosen pixels were taken from the material supplied with a PhD thesis in biology [59].
7.6.2 Evaluation criterion
For a visualisation of the results on a single movie, see Figures 7.28 and 7.29:
The domain expert has provided 13 pixel positions (green squares) corres-ponding to glomeruli that could be identified. As there can be more signals in a movie, e.g. caused by artifacts or glomeruli that could not be identified, all algorithms selected c= 30 pixels (magenta triangles).
The figures show the positions of pixels selected by the algorithms and by the domain expert superimposed onto a correlation image (Section 4.3.1) of the movie that was used to support manual selection. Indeed, the manually selected pixels fit well to the structures visible on the correlation image (Figures 7.28 and 7.29).
182 7 Empirical evaluation: NNCX andConvex cone
ConvexCone SiVM
local_NNCX CX_D
human expert algorithm shortest distance
Fig. 7.28.Columns selected by a human expert (c= 13 glomerulus positions) and by the respective CX algorithm (c= 30 pixels). The underlying movie is a calcium imaging recording from the honeybee AL [59] and pixels are superimposed onto a correlation image of the movie that was used by the expert for manual selection (Section 4.3.1). This figure is continued in Figure 7.29.
Yellow lines indicate the distance dr from a manually selected pixel pmanualr (green square) to the closest pixel that was selected by the respective algorithm,palgorithml (magenta triangle). The distancedris computed as
dr:= argminl
pmanualr −palgorithml
(7.8)
, and theprare vectors of x-and y coordinates in the image plane. Allpmanualr (green squares) are matched, but not necessarily to a unique palgorithml (magenta triangle). Multiple assignments of the same palgorithml to different pmanualr are indicative of a ”pixel of interest” not found by the algorithm and will result in large distances (yellow lines).
7.6 Concordance with manually selected points of interest 183
humanXexpert algorithm shortestXdistance
a,XGreedySpaceApproximation b,XLeverageScoreSampling
c,Xk-medoids
Convex SiVM Leverage
k-CX_D local_
Approx.
Greedy
Sampling 0
10
distancesX=XXXXXXX,X
d)
Cone Space
NNCXScore medoids N=1Xmovie,X13Xdistances
Fig. 7.29. a) Second part of Figure 7.28. Columns selected by a human expert (c= 13 glomerulus positions) and by the respective CX algorithm (c= 30 pixels).b) Median and (bar height) and median absolute deviation (whiskers) of the distances dr (yellow lines, Equation 7.8) in Figures 7.28, 7.29.
7.6.3 Example: Pixel selection on a single movie Performance of extreme vector methods
Convex cone achieved the best match to the domain expert solution, as visualised in Figures 7.28 and 7.29, and as measured by thedr:Convex cone was the only algorithm that achieved consistently low distances (low median and median deviation) to the manually chosen pixels (Figure 7.29d).
The other extreme vector algorithm, SiVM, performed also well with respect to choosing pixels close to the manually selected points, however, it again suffered from redundancy. For example,SiVMselected multiple pixels from the same glomerulus or region in the correlation image (Figure 7.28), which then led to other pixels being missed in a sample of size 30.
184 7 Empirical evaluation: NNCX andConvex cone
As a result, SiVM achieved a similarly low median distance, however with a large median deviation (Figure 7.29d) caused by the pixels that were missed.
VolumeSampling could not be performed as it terminated early due to numerical problems (see Section 5.4.1).
Performance of central vector methods
The performance of the central vector methods, in particular GreedySpace-Approximation, suffered less from few large distances, but rather from accumulating many smaller or medium distances.
As visible in Figure 7.29, GreedySpaceApproximationand LeverageScore-Sampling did cover the glomerulus areas on the correlation image, but especially GreedySpaceApproximation had also a tendency to select pixels from areas between the glomeruli or from the fringes of the glomeruli, instead of selecting them from the spatial middle of the glomeruli that was preferred by the domain expert. Consequently, both algorithms consistently achieved higher distancesdrto the manual solution (Figure 7.29d).
CX Dperformed better than GreedySpaceApproximationand Leverage-ScoreSampling, but it also selected multiple pixels from the same glomerulus or from the area between two glomeruli (Figure 7.28).
Performance of k-medoidsand local NNCX
Similar results were obtained for k-medoids and local NNCX that both selected pixels from the glomeruli, but not from the spatial middle of the glomeruli, and with a tendency to select (mixed signal) pixels from areas of overlap between glomeruli (Figures 7.28 and 7.29), also resulting in a larger median distance (Figure 7.29d).
7.6.4 Summary statistics for all movies
Generalising the example above (single movie), the histograms in Figure 7.30a summarise all distancesdr(yellow lines) pooled over all 12 movies. Depending on the movie, the domain expert selected different numbers of pixels (approx.
10-25). All algorithms selectedc= 50 pixels from each movie.
Both,Convex cone andSiVM accumulated many small distances, leading to histograms peaked at the left, whereSiVM, as in the example above, also caused a number of larger distances (Figure 7.30a), which is indicative of pixels missed completely. CX Dhad a distance distribution that was also peaked at the left, although not as pronounced as forConvex coneand also with a ”tail”
of larger distances at the right. The other column selection algorithms had flatter distance distributions with an approximately equal number of smaller and larger distances (Figure 7.30a).
7.6 Concordance with manually selected points of interest 185
0 distance 0 distance 40
20
0 distance 0 distance 40 0 distance 40
40
Fig. 7.30. a) Histograms of distances dr (yellow lines, Equation 7.8) achieved by the respective algorithm (c = 50). Distances dr are pooled across allN = 12 movies. b) Sum of the distances, P
rdr (sum of the yellow lines), in each of N=12 movies: Displayed as median (bar height) and median absolute deviation (whiskers). Convex Coneachieved a significantly lower summed distance than the other algorithms. Statistics: Kruskal-Wallis rank sum test (p = 3.587∗10−6), followed by post-hoc pairwise Wilcoxon rank sum tests with Holm correction: For Convex Cone, all correctedp <0.0044.
186 7 Empirical evaluation: NNCX andConvex cone
While the histograms visualise the distribution of distances across all movies, Figure 7.30b makes a statement about individual movies. The im-pression from Figures 7.28 and 7.29 was that Convex cone accumulates the smallest summed distance P
rdr (smallest summed line length) to the manual solution for a particular movie. Figure 7.30b confirms that this holds consistently across the 12 movies: Convex cone had a significantly smaller summed distance than the other algorithms, i.e. it resulted in the best approximation to human pixel selection across movies.
7.6.5 Settingc on imaging movies
As the evaluation criterion measured the shortest distance to each manually selected pixel (Equation 7.8), increasingcfor the algorithms usually decreased the summed distanceP
rdr(summed length of the yellow lines). Thus, higher c should be better up to the point at which all manually selected pixels have been discovered by the algorithm.
Figure 7.31 shows mean (standard deviations in Table 7.12) summed distances across all movies and for different values of c: For each movie, all summed distances were scaled by dividing through the value forConvex cone (c= 10). Then, distances for all movies were averaged.
Indeed, summed distances decreased with rising c for almost all algorithms (Figure 7.31). The only exception to the general distance decrease was local NNCX that, while performing well for small c, exhibited a distance increase for c > 30, probably because larger column sets were harder to optimise for the greedy heuristic. Recall further that the sparseness score of local NNCXdepended onc(Figure 7.22), with lower sparseness for smallc, suggesting that generally the choice ofcinfluences the results of local NNCX.
Convex cone achieved low distances already for small c and decreased its distance further than the other algorithms with rising c. A plateau was reached at around c = 40−50 (Figure 7.31), when also standard deviations were low (Table 7.12), i.e. the result was stable across movies. The plateau at c = 40−50 suggests that this is good choice for the parameter c, at least for this dataset (the number of signals/glomeruli can vary depending on experimental parameters). The exact value for c is not a critical parameter forConvex conethat computes a nested set of columns, which is in contrast tolocal NNCX that computes a newc-subset for eachc.
7.6 Concordance with manually selected points of interest 187
c k-medoids
ConvexCone SiVM CX_D
GreedySpaceApproximation LocalNNCX
LeverageScoreSampling
10 20 30 40 50
1.2
0.4
sum0of0distances0f0000000(0
Fig. 7.31. Mean of the summed distancesP
rdr (mean sum of the yellow lines, Equation 7.8) across all 12 movies for varying c. Before averaging, distances were divided by the value for (Convex cone,c= 10). The lowest mean summed distance achieved byConvex coneis marked by a straight, red line.
Convex
Cone SiVM CX D
Greedy Space Approx.
local NNCX
Leverage Score
Sampling k-medoids c = 10 0 0.185 0.286 0.153 0.148 0.167 0.25 c = 20 0.108 0.101 0.204 0.096 0.099 0.117 0.12 c = 30 0.074 0.106 0.163 0.087 0.069 0.094 0.073 c = 40 0.055 0.117 0.123 0.082 0.098 0.078 0.069 c = 50 0.05 0.12 0.11 0.052 0.075 0.079 0.071
Table 7.12.Standard deviation of the summed distancesP
rdr(sum of the yellow lines, Equation 7.8) across all 12 movies for varyingc.
188 7 Empirical evaluation: NNCX andConvex cone