Tracing of simulated structures

To develop an algorithm for the tracing of filamentous structures, suitable criteria for the de-termination of a filament’s position and direction need to be identified. 1D depletion patterns yielding a one-dimensional resolution increase seem to be the method of choice for imaging a fil-amentous structure, which itself only shows a FWHM below the diffraction limit perpendicular to the structure direction. To verify their suitability, 1D STED images of a ring-like structure are simulated for 1D depletion patterns of various orientations (see figure 4.14(a-c,e-g)). By the choice of a circle as underlying structure, all possible structure orientations are considered, enabling a comprehensive analysis of suitable criteria. All simulations within this subsection are performed with self-written Matlab(MathWorks, USA) code.

For an on-line adaption of the scan pattern, only the signal acquired at previous scan

posi-0 52 (b) (c) (d)

(a)

1 6

(e) (f) (g) (h)

0 240

0 52

0 52 0 52

0 52 0 52

Figure 4.14: Analysis of suitable criteria for determination of a filament’s position and direction.

(a-c,e-g) For a pixel size of 10 nm, the resulting image of a circular structure with a width of two pixels is calculated for six equally spaced 1D depletion pattern orientations for a resolution enhancement of k= 8. The scale bar indicates a length of 500 nm. (d) For each pixel, the variance of the intensity over the pattern orientations is calculated and displayed. (h) Analogously, the pattern direction yielding the maximum signal count is depicted for each pixel, with 1 denoting the direction yielding an angle of 60^◦ with they-axis (as in (a)) and 6 denoting the orientation enclosing an angle of−90^◦ with they-axis (as in (g)). Note that a mask (gray) defined by the variance (d) is applied to the data.

tions can be taken into account for the decision algorithm. Moreover, no complex analysis or reconstruction algorithms can be employed due to limited processing power on the FPGA.

Furthermore, for being a valuable tool for live-cell imaging of moving structures, the decision has to be taken in the order of the pixel dwell time and the algorithm needs to be robust against possible fluctuations in the total signal count.

For determining the scan pattern, both the structure’s position and its orientation need to be identified on a pixel basis, i.e. taking into consideration only the signal acquired on a single pixel for different depletion pattern orientations. For recognizing the structure’s position, the dependence of the signal’s variance on the depletion pattern orientation has been identified as a suitable criterion. Applying this criterion on a big field of view yields the results depicted in figure 4.14(d). On each pixel, the resulting signal count for all six pattern orientations is taken into account and the variance of those six values is calculated. Obviously, the variance is high close to the structure, while it is low in the absence thereof. For a filament tracing algorithm, this implies the necessity of scanning a line roughly perpendicular to the structure, since this yields the most pronounced peak of the variance. The maximal variance along this line then indicates the structure’s position.

The direction of a filamentous structure can be determined directly by comparing the signal on a single pixel obtained for the different pattern orientations. Therefore, the orientations are numbered as 1 to 6 with n denoting the orientation enclosing an angle of 90^◦ −n·30^◦ with they-axis. The orientation yielding the maximal signal count for each pixel is displayed in figure 4.14(h). For a better visualization, the data is masked such that the resulting value is set to zero wherever the variance at that position is less than 30% of the overall maximal variance (cf. figure 4.14(d)). It is clearly evident that the chosen pattern orientation can actually well approximate the real direction of the structure under analysis.

Hence, the filament’s position and direction can be uniquely determined by first identifying the structure’s position, employing the signal’s variance, and then determining the pattern orientation yielding the maximal signal at the position of maximal variance.

Based on these results, a tracing algorithm can now be developed. For biological samples, an upper limit on the structure’s curvature can be assumed, implying that a filament’s di-rection typically does not change drastically within the course of some pixels. Therefore, the filament’s direction does not need to be determined on a line-wise basis. Instead, small blocks of lines are scanned, with the decision algorithm conducted at the end of each block, and the number of lines per block is chosen depending on the maximal curvature.

Hence, for a given starting position and direction, a scan of a pre-defined number of lines of a certain length, i.e. pixel number, is conducted (cf. also figure 3.17) on a pre-defined scanning grid. The orientation of these lines is chosen according to the direction of the structure: For a horizontally oriented structure, i.e. pattern orientation 6 (see figure 4.15(f)), the lines are

(a) (b) (c) (d) (e) (f)

Figure 4.15: Sketch of the scan pattern for the tracing algorithm. For a structure direction corre-sponding to an optimal depletion pattern orientation of 1 to 6 (cf. figure 4.14(h)), the line scan for the tracing algorithm is chosen as indicated in (a-f) based on a pre-defined scanning grid. A displacement of the lines inxandy ensures the best possible approximation of the structure’s direction.

initially chosen vertically. Each line is displaced by one pixel in x-direction relative to the previous line, as schematically illustrated in figure 4.15(f). Analogously, horizontal lines are chosen for a vertically oriented structure, as indicated in figure 4.15(c). For a structure ori-entation yielding the depletion pattern oriori-entation 1 optimal, as shown in figure 4.15(a), the lines are not purely displaced inx-direction, but also iny-direction in order to yield an optimal scanning of the structure. Therefore, every two lines experience a shift iny-direction by one pixel, and analogously in x-direction for the pattern orientations 2 (see figure 4.15(b)) and 4 (see figure 4.15(d)).

After a pre-defined number of lines has been scanned, the signal counts for the last scanned line are evaluated to determine the current position and direction of the filament. The pixel yielding the maximal variance between pattern orientations is identified as the new position of the structure. For that pixel, the pattern orientation resulting in the highest signal count is determined and identified as the new structure’s direction. The scan pattern for the next block can now be fully determined according to figure 4.15 and the scan is continued.

The tracing algorithm is first tested inMatlabon a simulated filamentous structure. A reso-lution enhancement ofk= 8 is chosen for the simulated PSFs, necessitating at least 6 pattern orientations when employing Richardson-Lucy deconvolution. The corresponding simulated images are displayed in figure 4.16(a-c,e-g). The data is reconstructed by both maximum-value reconstruction (see figure 4.16(d)) and Richardson-Lucy deconvolution (see figure 4.16(h)), with the former showing slight artifacts at the turns of the filaments due to the insufficient sampling with pattern orientations. Figure 4.16(i-k,m-o) show the images acquired by the tracing algo-rithm for the six pattern orientations. Clearly, the inner ring of the structure is successfully detected and the scanning is limited to a band which fully includes the filament. As seen in comparison to e.g. figure 4.16(h), also turns of the structure are correctly recognized. Note that the algorithm is stopped before the starting position is reached again. The data can be reconstructed employing either maximum-value reconstruction or Richardson-Lucy deconvolu-tion, with the results presented in figure 4.16(l) and (p), respectively. Both algorithms yield a good estimation of the structure as compared to figure 4.16(d) and (h). Nonetheless, at the

0 42 (b) (c) (d) (a)

(e) (f) (g) (h)

(i) (j) (k) (l)

0 213

(m) (n) (o) (p)

0 42

0 41

0 42

0 42 0 42

0 42

0 41 0 41

0 41

0 41 0 41

0 42

0 56

0 369

start

Figure 4.16: Results of the first tracing algorithm on a simulated filamentous structure. A filamen-tous structure is simulated for a pixel size of 5 nm with a structure width of three pixels. (a-c,e-g) Resulting images of the simulated structure for six equally spaced 1D depletion patterns for a resolution enhancement ofk= 8. The scale bar indicates a length of 500 nm. (d) Maximum-value reconstruction of the data shown in (a-c,e-g). Negative pixel, resulting from discontinuities in the assembled OTF (cf.

[Krü17]), are set to 0 for the visualization. (h) Richardson-Lucy deconvolution of the same data with 10 iteration steps andαRch = 0.0001. (i-k,m-o) Images acquired by the tracing algorithm for the six depletion pattern orientations. Only a narrow band of 31 pixel, corresponding to 155 nm, around the filament is scanned. Scan position and direction are adapted every 5 lines and 120 steps are performed in total. (l) Maximum-value reconstruction of the data (i-k,m-o). Artifacts arise due to black corners in the scan at the turns of the filament (highlighted by green arrows). (p) Richardson-Lucy deconvolution (10 iteration steps,α_Rch= 0.0001) for the same data. Also here, the filaments seem incomplete at the positions of the turns.

turns of the filaments, black areas are visible where the filaments are incomplete (indicated by green arrows in figure 4.16), especially in the maximum-value reconstruction. These areas are already visible in the raw data and result from an incomplete scanning of the filament at the turn: If the orientation of the lines for the scan is changed from horizontal to vertical or vice versa, parts of the filament are not scanned by this version of the algorithm. This can be avoided as outlined in the following.

To optimize the algorithm, the change of a horizontal to a vertical line scan is detected during the decision process. Whenever this occurs, the scan is not continued with the newly deter-mined scan position and direction, but instead it is still continued in the same direction as before for a specified number of additional lines. Nevertheless, the newly determined para-meters are saved and employed as starting position in the next iteration step. The results of this improved tracing algorithm on simulated data are shown in figure 4.17(a-c,e-g) for the six depletion pattern orientations. The additional line scans at turns of the filament are clearly visible and result in a more complete reconstruction, as seen from figure 4.17(d) and (h) for the maximum-value reconstruction and the Richardson-Lucy deconvolution, respectively. Both re-construction techniques yield a good approximation of the structure. At the intersection points, the maximum-value reconstruction even allows an approximation of the initial direction of the filaments outside the scanned region.

0 41 (b) (c) (d)

(a)

(e) 0 41 (f) (g) (h)

0 41

0 41 0 41

0 41 0 49

0 333

start

Figure 4.17: Results of the optimized tracing algorithm on a simulated filamentous structure. The same structure as in figure 4.16 is simulated. The tracing algorithm is optimized to avoid the formation of left-out corners in the scan by increasing the number of lines before turns. The results for the six depletion pattern orientations for otherwise same parameters as in figure 4.16 are displayed in (a-c,e-g).

(d) shows the maximum-value reconstruction and (h) the Richardson-Lucy deconvolution (10 iteration steps,α_Rch= 0.0001) for the data shown in (a-c,e-g).