Visual classification of complicated plaques based on multidimensional image fusion
A. Hennemuth1, A. Harloff2, T. Spehl2, N. Pavlov2, O. Friman1, D. Paul2, D. v. Elverfeldt2, C. Kuehnel1, S. Wirtz1, H. Goebel3, J. Mannheim4, H. K. Hahn1, B. Pichler4, J. Hennig2, M. Markl2
anja.hennemuth@fraunhofer.mevis.de
1Fraunhofer MEVIS, Bremen,2University Hospital Freiburg,
3University Hospital Cologne,4University Hospital Tuebingen
Abstract: Multi-contrast MRI has shown to be suitable for plaque analysis, which is important for the risk assessment of internal carotid artery stenoses. The purpose of this work was the development of software prototypes, which support the develop- ment of methods for the classification and exploration of plaque-morphology in 3D and allow for a comparison with histological images and CT images. Complex ex- vivo plaque specimen were evaluated at a high field 9.4T MRI system with different contrasts (T1-fatsat-GRE, T2*-GRE, T2-RARE, PD-RARE) and high resolution (ca.
100µm3). Plaque-images were processed to enhance specific plaque-components and combined into RGB-images. The resulting images were interactively explored in cou- pled views of volume-rendering and user-defined MPRs. µCT images were also pre- processed for visualization. The user could then explore corresponding image regions in MR images, histological images and CT images. Initial results showed a benefit for the assessment of plaque-constitution and 3D-morphology.
1 Introduction
High-grade internal carotid artery (ICA) stenoses are a leading source of ischemic stroke and the assessment of the risk factors is a crucial task. The plaque constitution is an important risk factor [SHT+07]. Especially complicated type VI plaques according to the American Heart Association (AHA) classification contain thrombi, acute or old in- traplaque hemorrhage and/or ulceration of the fibrous cap [SCD+95]. While CT gives a reliable detection and quantification of calcifications [dWOM+06], the analysis of histo- logical slices is the goldstandard for the analysis of other plaque constituents.
Previous studies have shown that MRI exhibits excellent softtissue contrast and can be used to distinguish major components of carotid atherosclerotic plaque [YKF+02, CFU+05, FMS+08]. In addition, T1, T2 and proton-density(PD) weighted sequences, T2* and dif- fusion weighted sequences have been evaluated for plaque characterization [QRV+07].
For the analysis of multicontrast MRI of plaques, there exist several approaches and most of them focus on the segmentation of plaque images acquired with T1, T2 and PD weighted images [XHY02, AvdGW+04, ISM+04, CBH+06, ASH+07, KBV+09, LXF+06, SGL+08]. A major problem in the development of intensity-based classifica- tion algorithms for visual or automatic image analysis is the dependency of the intensity
distribution upon the imaging parameters. Thus, some approaches train classification al- gorithms using reference segmentations from histological slices and CT images to adapt them to a given parameter set [LXF+06, CBH+06, ASH+07]. Other approaches based on fuzzy clustering start with a representative sample [ISM+04] or a priori-knowledge of the quantitative T2 map [SGL+08].
For the training of classificators and the validation of results, the MR images have to be compared with the goldstandards CT (for calcifications) and/or histology. Histological images normally consist of a small set of slices and the orientation may differ from the MR image slices. Furthermore, the preparation process leads to deformations and rup- tures in the loosely connected tissue. An accurate matching of the images is therefore difficult. In previous studies, experts visually inspected the data and transferred the seg- mentation of the histology images to the region of the MR image they assumed as matching [ISM+04, ASH+07]. Clarke et al. apply Delauney triangle nonlinear registration to match
Figure 1: Histological images with deformations from the preparation procedure
the histological images with the MR images for an automatic per voxel comparison of the classification [CBH+06].
For the alignment of MRI and CT images linear registration is applied [DSC+06, CBH+06].
Dey et al. compare in-vivo CT and T1-weighted MR images and thereto apply mutual in- formation based linear registration to the segmented vessels. Clarke et al. also use linear registration to alignµCT and MR images of ex-vivo plaques. Fig. 2 shows an example of aµCT image and a corresponding MR image.
Previous studies have either focussed on the segmentation of ex-vivo images or on the analysis of in-vivo images. Furthermore, different tools like Matlab, ImageJ and ENVI are applied in the analysis process.
The aim of the present work is to provide a set of software assistants that support the analysis and comparison of histological images, CT images and multicontrast MRI ex- vivo and in-vivo. On the one hand, the common software environment should facilitate the cooperative analysis of study data by pathologists, radiologists and neurologists, on the other hand, this software should provide a framework for the evaluation of imaging sequences and classification algorithms for ex-vivo as well as in-vivo image data.
Figure 2: Examples of an ex-vivo T1-weighted MR image and the correspondingµCT image. The field of view and the orientation differs.
Contrast Sequence Res. [mm3] TE / TR [ms] Flip angle Turbo factor Averages
T1 GRE 0.1 2.4/13.4 20 N/A 12
T2* GRE 0.1 7.9/12.8 20 N/A 10
T2 RARE 0.1 40/2500 90-180 16
PD RARE 0.1 9/1500 90-180 8
Table 1: Pulse sequence parameters for multi-contrast 3D ex-vivo plaque imaging using an ultra-high field small animal system (9.4T).
2 Data
To date, in an ongoing study, plaque specimen of patients undergoing carotid endarterec- tomy for ICA stenosis≥70%(local degree of stenosis) are evaluated. Plaques are fixed in 4% formalin for≥24 hours after surgical resection, transferred into tubes containing free oil (1,1,2-Trichlorotrifluoroethane) and evaluated at a high field 9.4T MRI system (Bruker BioSpin 94/20, Ettlingen, Germany). The MRI protocol provides 3D imaging with a spa- tial resolution of ˜100µm3including T1-fatsat-GRE, T2*-GRE, T2-RARE, and PD-RARE imaging (see Table 1). Total scan time is 10 hours. After MR scanning the plaque speci- men are scanned with aµCT Scanner (SiemensµCAT II) before EVG-coloured histolog- ical images of single slices are produced. The in-vivo images were acquired prior to the intervention with a 3T MRI scanner (Siemens TrioTim, Erlangen, Germany). They con- tain a Time-of-Flight(TOF) image volume (0.52×0.52.×1.0mm3)and T1-, T2-, T2*- and PD-weighted images acquired at lower resolutions (ca. 0.54×0.54×4.0).
3 Methods
To support the analysis of the study data, two software prototypes are being developed. The first one supports the manual classification of the histological slices by the pathologist.
The user can interactively define contours and label the according regions. The second prototype focusses on the processing of the volume images from CT and MRI and the exploration of the spatial and contentual correspondances of the different images. Figure 3 shows an overview of the main functionalities of the actual software.
Figure 3: Overview of the image data and prototype functionality for the analysis of the ex-vivo plaques. The histological images are classified manually using a special prototype. CT and MR images are preprocessed within the same prototype before they can be explored separately or fused in 3D. The interactive definition of arbitrary slice planes in 3D supports the detection of the CT and MR image regions that correspond with the histological slices.
The plaque analysis software is implemented within the image processing and visualiza- tion R&D platform MeVisLab [BVUW+07]. CT and MR images are preprocessed to remove non-plaque structures and background noise.
3.1 Preprocessing
In the CT images, the cylinder that contains the plaque is clearly visible (Fig. 2). To restrict the visualization to the interesting image region, the cylinder is removed automatically as shown in Fig. 4. First, an automatic thresholding is applied. The threshold is determined by the analysis of the histogram of a central image slice. The cylinder component is then detected. The mask that results from filling the cylinder component is then used to mask the original image.
Figure 4: Removal of surrounding structures fromµCT image. The leftmost image shows the ac- quired image with the plaque and the surrounding structures. The middle image shows the result of the component analysis with the plaque and the containing cylinder identified as one component.
The rightmost image shows the masked plaque image, where the structures surrounding the plaque are removed.
To prepare the MR images for the combined visualization, an initial semi-automatic back- ground suppression is applied. Thereto the available contrasts are combined into a max- imum image, which is then processed with an Otsu Threshold [Ots79] and a subsequent morphological closing. Based on this pre-segmentation the contours of the outer border and the lumen are detected to further distinguish dark plaque regions and lumen through the contour lengths (Fig. 5). The resulting contours can be corrected interactively by the users.
3.2 Visual Exploration of Multicontrast MR Images
Many of the approaches described in Section 1 use exactly three MR contrasts and it is common to fuse these contrast images into an RGB image for further processing and com- parison with histological slices[ISM+04, ASH+07]. This approach is extended here. The user can select the contrast for each channel and window each color channel separately to get the most meaningful representation of the fused image (Fig. 6).
Previous studies have given estimates for the relative appearance of specific plaque com- ponents at different contrasts [FMS+08]. This knowledge can be applied to suppress or enhance components through arithmetic combinations. Figure 8 shows such a fusion that
Figure 5: Separation of plaque and background based on the maximum image of the given MR contrasts. After the application of thresholding and morphological closing plaque and background are separated by the detection of the contours.
combines the PD-weighted and the T2-weighted images to enhance the fibrous cap and the lipid rich/necrotic core regions.
The 3D and MPR views are coupled: MPRs are defined by moving and rotating the cor- responding lines in the intersecting viewing planes and can be shown in the 3D volume rendering for orientation [LKP06]. The current combination image can also be overlaid onto the original images, which are shown in the 2D viewers on the left side (Fig. 9).
3.3 Comparison of Histological Images and Image Volumes
To compare the volumetric CT and MR images with the histological slices, it is important to find the corresponding transsection plane. The user can reproduce the slice position by adjusting the MPR in the 3D viewing panel. The 2D image panel on the left can either show 2D views of the volume images or the EVG-colored histological slice images and the manual segmentation results of the pathologist. To facilitate the interactive position matching the histological slice can also be rotated (Fig. 6).
3.4 Analysis of in-vivo Plaque Images
In addition to the low resolution due to the shorter scanning time, in-vivo plaque images are affected by patient motion. Thus, a combined analysis of the different MR contrasts requires a prior registration. Thereto we apply a 3D registration based on normalized mu- tual information and affine transformations. To identify the region of interest and quantify the vessel narrowing caused by the plaques, the vessels are first segmented in the TOF image with a region growing approach [BRL+04]. The result of this segmentation can then be used to identify the interesting image regions for the exploration and the compar- ison with the ex-vivo imaging results. Figure 7 shows a visualization result for registered
Figure 6: Comparison of the histological image and the corresponding image region in the RGB image that results from the fusion of the PD-weighted, T2-weighted and T2*-weighted ex-vivo MR images. The user can rotate the histological image and define the corresponding slice plane by moving and rotation the slice markers in the orthogonal views.
color-combined in-vivo images.
4 Results and Discussion
In an ongoing study the described tools have been applied to three datasets. Histological slices have been classified successfully by the pathologist and results have been transferred to the neurological department, where the study results are evaluated.
The visual exploration of the multicontrast MR images resulted in a quicker apprehension of the plaque constitution than the conventional side-by-side comparision of the separate image slices. The left part of figure 8 shows the results of a high resolution multi-contrast plaque acquisition as viewed in the standard analysis procedure. By comparing signal intensities in T1, T2, T2* and PD weighted images, different plaque components such as calcification, hemorrhage, and lipid rich/necrotic core (LR/NC) are identified slicewise.
Applying the new analysis tool, the same data was used to create novel contrasts, as shown on the right side in Fig. 8 multiplying T2 and PD contrast of the spatially registered plaque data. Note, that in this case, contrast manipulation provides a clearer delineation of the fibrous cap and the LR/NC compared to the standard view in left part of Fig. 8.
The use of different color channels for the combination of three contrasts into a single image is shown in Fig. 9 for another plaque specimen. The combination of T1 (red color channel), T2 (green color channel) and PD (blue color channel) illustrates the potential for improved identification of plaque components. Particularly fibrous tissue (hyperintense in
Figure 7: Fused visualization of in-vivo multicontrast MR images. The vessels are segmented in the TOF image, which covers a larger region than the other images. The plaque is clearly visible on the RGB image that stems from the T1-, T2- and PD-weighted images.
T2 and thus green in Fig. 9), calcification (signal void), and necrosis (hyperintense in T1 and thus red in Fig. 9) can clearly be identified.
An expert determined the image regions corresponding to the histological slices in the MR volumes. To finally transfer the classification results of the pathologist, a non-rigid registration as proposed by Clarke et al. [CBH+06] will be necessary to compensate the deformations from the preparation of the histological slices (Fig. 1).
The analysis of the in-vivo images has not been adressed in the study yet. However, first tests with the given datasets indicate that the methods developed with the ex-vivo image data can be transferred to in-vivo images.
5 Conclusions
In the previous sections we presented a prototypical software for multi-modal inspection of carotid plaque specimens. The software prototype offers functionalities for interactive classification of histological images, texploration of single MR contrasts or images fused via arithmetic operations or color channels. Furthermore, an interactive alignment of his- tological images and volumetric images is supported. In an ongoing study, first datasets have been processed with the software. Initial results indicate the potential for improved delineation of specific plaque components. 3D visualization of the resulting contrast or color maps allow to explore and understand the complex 3D plaque morphology. Future work will focus on the development and evaluation of automatic analysis methods for multicontrast MR images ex-vivo and in-vivo.
Figure 8: Left:Original multi-contrast images of a complex ex-vivo plaque specimen with calcifi- cation, a necrotic core including old hemorrhage, and a fibrous cap separating the core and vessel lumen. Right: 3D ex-vivo plaque analysis with multiplied T2 and PD contrast highlighting fibrous tissue. Note, that the fibrous cap is considerably enhanced and easier to identify than in the standard view shown in Fig. 1. 3D coverage facilitates the evaluation of thickness and extent of cap and plaque core within the entire specimen.
References
[ASH+07] Russell W Anderson, Christopher Stomberg, Charles W Hahm, Venkatesh Mani, Daniel D Samber, Vitalii V Itskovich, Laura Valera-Guallar, John T Fallon, Pavel B Nedanov, Joel Huizenga, and Zahi A Fayad. Automated classification of atheroscle- rotic plaque from magnetic resonance images using predictive models. Biosystems, 90(2):456–466, 2007.
[AvdGW+04] I. M. Adame, R. J. van der Geest, B. A. Wasserman, M. A. Mohamed, J. H C Reiber, and B. P F Lelieveldt. Automatic segmentation and plaque characterization in atherosclerotic carotid artery MR images.MAGMA, 16(5):227–234, Apr 2004.
[BRL+04] Tobias Boskamp, Daniel Rinck, Florian Link, Bernd K¨ummerlen, Georg Stamm, and Peter Mildenberger. New vessel analysis tool for morphometric quantification and visualization of vessels in CT and MR imaging data sets.Radiographics, 24(1):287–
297, 2004.
[BVUW+07] I. Bitter, R. Van Uitert, I. Wolf, L. Ibanez, and J.-M. Kuhnigk. Comparison of Four Freely Available Frameworks for Image Processing and Visualization That Use ITK.
13(3):483–493, 2007.
[CBH+06] Sharon E Clarke, Vadim Beletsky, Robert R Hammond, Robert A Hegele, and Brian K Rutt. Validation of automatically classified magnetic resonance images for carotid plaque compositional analysis.Stroke, 37(1):93–97, Jan 2006.
[CFU+05] Baocheng Chu, Marina S Ferguson, Hunter Underhill, Norihide Takaya, Jianming Cai, Michel Kliot, Chun Yuan, and Thomas S Hatsukami. Images in cardiovascular medicine. Detection of carotid atherosclerotic plaque ulceration, calcification, and thrombosis by multicontrast weighted magnetic resonance imaging. Circulation, 112(1):e3–e4, Jul 2005.
[DSC+06] Damini Dey, Piotr Slomka, Daisy Chien, David Fieno, Aiden Abidov, Rola Saouaf, Louise Thomson, John D Friedman, and Daniel S Berman. Direct quantitative in
Figure 9: Multi-contrast plaque analysis tool. Different contrasts are displayed via the RGB color channels: T1 (red), T2 (green) and PD (blue).
vivo comparison of calcified atherosclerotic plaque on vascular MRI and CT by mul- timodality image registration.J Magn Reson Imaging, 23(3):345–354, Mar 2006.
[dWOM+06] Thomas T de Weert, Mohamed Ouhlous, Erik Meijering, Pieter E Zondervan, Jo- hanna M Hendriks, Marc R H M van Sambeek, Diederik W J Dippel, and Aad van der Lugt. In vivo characterization and quantification of atherosclerotic carotid plaque components with multidetector computed tomography and histopathological correlation.Arterioscler Thromb Vasc Biol, 26(10):2366–2372, Oct 2006.
[FMS+08] Sebastiano Fabiano, Stefano Mancino, Matteo Stefanini, Marcello Chiocchi, Alessandro Mauriello, Luigi Giusto Spagnoli, and Giovanni Simonetti. High- resolution multicontrast-weighted MR imaging from human carotid endarterectomy specimens to assess carotid plaque components.Eur Radiol, 18(12):2912–2921, Dec 2008.
[ISM+04] V.V. Itskovich, D.D. Samber, V. Mani, J.G.S. Aguinaldo, J.T. Fallon, C.Y. Tang, V. Fuster, and Z.A. Fayad. Quantification of human atherosclerotic plaques us- ing spatially enhanced cluster analysis of multicontrast-weighted magnetic reso- nance images. Magnetic Resonance in Medicine : Official Journal of the Society of Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine, 52(3):515–523, 2004.
[KBV+09] Christof Karmonik, Pamela Basto, Kasey Vickers, Kirt Martin, Micheal J Reardon, Gerald M Lawrie, and Joel D Morrisett. Quantitative segmentation of principal carotid atherosclerotic lesion components by feature space analysis based on mul- ticontrast MRI at 1.5 T.IEEE Trans Biomed Eng, 56(2):352–360, Feb 2009.
[LKP06] F. Link, M. Koenig, and H.-O. Peitgen. Multi-Resolution Volume Rendering with per Object Shading. In L. Kobbelt, T. Kuhlen, and R. Westermann, editors,Vision Modelling and Visualization, pages 185–191, Berlin, Aachen, 2006. Aka.
[LXF+06] Fei Liu, Dongxiang Xu, Marina S Ferguson, Baocheng Chu, Tobias Saam, Norihide Takaya, Thomas S Hatsukami, Chun Yuan, and William S Kerwin. Automated in vivo segmentation of carotid plaque MRI with Morphology-Enhanced probability maps.Magn Reson Med, 55(3):659–668, Mar 2006.
[Ots79] Nobuyuki Otsu. A Threshold Selection Method from Gray-Level Histograms.IEEE Transactions on Systems, Man and Cybernetics, 9(1):62–66, January 1979.
[QRV+07] Ye Qiao, Itamar Ronen, Jason Viereck, Frederick L Ruberg, and James A Hamil- ton. Identification of atherosclerotic lipid deposits by diffusion-weighted imaging.
Arterioscler Thromb Vasc Biol, 27(6):1440–1446, Jun 2007.
[SCD+95] H. C. Stary, A. B. Chandler, R. E. Dinsmore, V. Fuster, S. Glagov, W. Insull, M. E.
Rosenfeld, C. J. Schwartz, W. D. Wagner, and R. W. Wissler. A definition of ad- vanced types of atherosclerotic lesions and a histological classification of atheroscle- rosis. A report from the Committee on Vascular Lesions of the Council on Arte- riosclerosis, American Heart Association. Circulation, 92(5):1355–1374, Sep 1995.
[SGL+08] B. Sun, D.P. Giddens, R.J. Long, W.R. Taylor, D. Weiss, G. Joseph, D. Vega, and J.N. Oshinski. Automatic plaque characterization employing quantitative and mul- ticontrast MRI. Magnetic Resonance in Medicine : Official Journal of the Society of Magnetic Resonance in Medicine / Society of Magnetic Resonance in Medicine, 59(1):174–180, 2008.
[SHT+07] T. Saam, T.S. Hatsukami, N. Takaya, B. Chu, H. Underhill, W.S. Kerwin, J. Cai, M.S. Ferguson, and C. Yuan. The vulnerable, or high-risk, atherosclerotic plaque:
noninvasive MR imaging for characterization and assessment.Radiology, 244(1):64–
77, 2007.
[XHY02] D. Xu, J.-N. Hwang, and C. Yuan. Segmentation of Multi-Channel Image with Markov Random Field Based Active Contour Model. J. VLSI Signal Process. Syst., 31(1):45–55, 2002.
[YKF+02] Chun Yuan, William S Kerwin, Marina S Ferguson, Nayak Polissar, Shaoxiong Zhang, Jianming Cai, and Thomas S Hatsukami. Contrast-enhanced high resolu- tion MRI for atherosclerotic carotid artery tissue characterization. J Magn Reson Imaging, 15(1):62–67, Jan 2002.
