Image-based tracking, quantification and exploration of cardiac dynamics

Exploration of Cardiac Dynamics

vorlegt von Diplom-Informatiker

Lennart Tautz

ORCID: 0000-0002-1758-6111

an der Fakult¨at IV - Elektrotechnik und Informatik der Technischen Universit¨at Berlin

zur Erlangung des akademischen Grades Doktor der Ingenieurwissenschaften

Dr.Ing. -genehmigte Dissertation

Promotionsausschuss:

Vorsitzender: Prof. Dr.-Ing. Olaf Hellwich Gutachterin: Prof. Dr.-Ing. Anja Hennemuth Gutachter: Univ.-Prof. Dr. med. Volkmar Falk Gutachterin: Dr. techn. Katja B¨uhler

Tag der wissenschaftlichen Aussprache: 30. Juni 2020

The heart is the central driver of blood circulation, which supplies the body with oxy-gen. Heart function is determined by a complex interaction between heart wall con-traction, heart valves and respiration. Cardiac diseases and disease progression man-ifest often regionally and diversely across patients. Modern clinical imaging allows the acquisition of image data for the detailed assessment of the state and dynamics of the heart muscle and the valves. Quantification and advanced analysis of these struc-tures require image processing and image-based modeling techniques to enable the patient-specific evaluation of heart function. The goal of this thesis is to investigate and implement methods for the regional analysis of motion patterns in the heart wall, segmentation and analysis of the mitral valve for quantification and simulation, and for the integrated visualization of time-resolved cardiac multi-parameter results.

The developed methods include efficient registration-based wall motion analy-sis in 3D tagged MRI and echocardiography images, interventricular septum motion analysis in cine MRI images time-resolved by cardiac and respiratory phase, semi-automatic segmentation of the mitral valve in 3D CT images combining user-defined landmarks and image information, automatic segmentation of the mitral valve in 4D echocardiography images by tracking a deformable model, regional subdivision of a mitral valve model for standardized reporting, and an interactive hierarchical explo-ration approach incorporating multiple parameters and the relevant temporal dimen-sions to make the quantitative results of analysis methods such as the previous ones accessible to medical experts.

All techniques have been evaluated on a variety of phantom and clinical data. The heart wall motion analysis performed with satisfactory accuracy and strain recovery in different modalities. The septum analysis produced consistent motion parame-ters, and showed the need for an appropriate visualization tool. The evaluation of the semi-automatic valve segmentation demonstrated an efficient method to define a valve model in image data with varying contrast and in the presence of pathologies. The automatic 4D valve segmentation produced satisfactory valve models in normal and pathological cases. The accuracy of the regional subdivision was comparable to the variation between user experts. The interactive exploration approach demon-strated a promising approach that was valued as helpful for the task. Remaining chal-lenges that have to be addressed in future work include the temporal consistency of wall motion analysis, and high inter-operator variation in mitral valve segmentation.

Das Herz ist das treibende Organ im Blutkreislauf, der den Körper mit Sauerstoff ver-sorgt. Die Herzfunktion wird durch ein komplexes Zusammenspiel zwischen Wand-kontraktion, Herzklappen und Atmung bestimmt. Erkrankungen des Herzens und ih-re Verläufe manifestieih-ren sich oft ih-regional und je nach Patient unterschiedlich. Mo-derne klinische Bildgebung ermöglicht die Aufnahme von Bilddaten für die detaillier-te Bewertung von Zustand und Dynamik von Herzmuskel und -klappen. Quantifizie-rung und weitergehende Analyse dieser Strukturen erfordert Methoden der Bildver-arbeitung und bildbasierten Modellierung, um patientenspezifisch die Herzfunktion bewerten zu können. Das Ziel dieser Arbeit ist die Untersuchung und Implementie-rung von Methoden zur regionalen Analyse von Bewegungsmustern in der Herzwand, zur Segmentierung und Analyse der Mitralklappe zur Quantifizierung und Simulati-on, und zur integrierten Visualisierung zeitaufgelöster kardialer multiparametrischer Ergebnisse.

Die entwickelten Methoden umfassen eine effiziente registrierungsbasierte Wand-bewegungsanalyse in 3D-Tagged-MRT- und Echokardiographie-Daten, die Bewegungs-analyse des interventrikulären Septums in Cine-MRI-Bildern mit zeitlicher Auflösung von Herz- und Atemphase, die semi-automatische Segmentierung der Mitralklappe in 3D-CT-Daten durch Kombination von benutzerdefinierten Landmarken und Bild-informationen, die automatische Segmentierung der Mitralklappe in 4D-Echokardio-graphie-Bildern durch Tracking eines verformbaren Modells, die regionale Untertei-lung eines Mitralklappenmodells zur standardisierten Dokumentation, und einen in-teraktiven hierarchichen multiparametrischen Explorationsansatz, der alle relevan-ten zeitlichen Dimensionen beinhaltet und damit die quantitativen Ergebnisse von Analysemethoden wie den vorgenannten für medizinische Experten zugänglich ma-chen kann.

Alle Verfahren wurden auf einer Vielzahl von synthetischen und klinischen Daten evaluiert. Die Herzwandbewegungsanalyse erbrachte zufriedenstellende Genauigkeit und Strainberechnung in verschiedenen Bildmodalitäten. Die Analyse des Septums ergab konsistente Bewegungsparameter, und zeigte den Bedarf an einem geeigneten Visualisierungswerkzeug auf. Die Evaluierung der semi-automatischen Klappenseg-mentierung demonstrierte eine effiziente Methode, um in Bilddaten mit wechseln-dem Kontrast und Pathologien ein Klappenmodell zu definieren. Die automatische 4D-Klappensegmentierung erbrachte zufriedenstellende Klappenmodelle in norma-len und pathologischen Fälnorma-len. Die Genauigkeit der regionanorma-len Unterteilung war ver-gleichbar mit der Variation zwischen menschlichen Experten. Der interaktive Explo-rationsansatz wurde als vielversprechender Ansatz und als hilfreich für die Aufga-be Aufga-bewertet. VerbleiAufga-bende Herausforderungen für zukünftige ArAufga-beiten umfassen die zeitliche Konsistenz der Wandbewegungsanalyse, und hohe Inter-Operator-Variation der Mitralklappensegmentierung.

The following publications are part of this thesis:

1. Tautz L, Hennemuth A, and Peitgen HO (2012) "Motion Analysis with Quadrature Filter Based Registration of Tagged MRI Sequences". In: Camara O, Konukoglu E, Pop M, Rhode K, Sermesant M, Young A (eds) Statistical Atlases and Computational

Models of the Heart. Imaging and Modelling Challenges. STACOM 2011. Lecture

Notes in Computer Science, vol 7085. Springer, Berlin, Heidelberg. https://doi. org/10.1007/978-3-642-28326-0_8 (Chapter 4)

2. Tobon-Gomez C, De Craene M, McLeod K, Tautz L, Shi W, Hennemuth A, Prakosa A, Wang H, Carr-White G, Kapetanakis S, Lutz A, Rasche V, Schaeffter T, Butakoff C, Friman O, Mansi T, Sermesant M, Zhuang X, Ourselin S, Peitgen HO, Pennec X, Razavi R, Rueckert D, Frangi AF, and Rhode KS (2013) "Benchmarking framework for myocardial tracking and deformation algorithms: An open access database".

Medical Image Analysis, Volume 17, Issue 6, Pages 632-648. Elsevier.https://doi.

org/10.1016/j.media.2013.03.008 (Chapter 5)

3. Tautz L, Hennemuth A, and Peitgen HO (2013) "Quadrature Filter Based Motion Analysis for 3D Ultrasound Sequences". In: Camara O, Mansi T, Pop M, Rhode K, Sermesant M, Young A (eds) Statistical Atlases and Computational Models of the

Heart. Imaging and Modelling Challenges. STACOM 2012. Lecture Notes in

Com-puter Science, vol 7746. Springer, Berlin, Heidelberg. https://doi.org/10.1007/ 978-3-642-36961-2_20 (Chapter 6)

4. De Craene M, Marchesseau S, Heyde B, Gao H, Alessandrini M, Bernard O, Piella G, Porras AR, Tautz L, Hennemuth A, Prakosa A, Liebgott H, Somphone O, Allain P, Makram Ebeid S, Delingette H, Sermesant M, D’hooge J, and Saloux E (2013) "3D Strain Assessment in Ultrasound (Straus): A Synthetic Comparison of Five Track-ing Methodologies". IEEE Transactions on Medical ImagTrack-ing, 32(9), pp. 1632-1646. IEEE.https://doi.org/10.1109/TMI.2013.2261823 (Chapter 7)

5. Tautz L, Feng L, Otazo R, Hennemuth A, and Axel L (2016) "Analysis of cardiac interventricular septum motion in different respiratory states". Proc. SPIE 9788,

Medical Imaging 2016: Biomedical Applications in Molecular, Structural, and Func-tional Imaging, 97880X. InternaFunc-tional Society for Optics and Photonics. https:

//doi.org/10.1117/12.2216240 (Chapter 8)

6. Tautz L, Neugebauer M, Hüllebrand M, Vellguth K, Degener F, Sündermann S, Wa-mala I, Goubergrits L, Kuehne T, Falk V, and Hennemuth A (2018). "Extraction of open-state mitral valve geometry from CT volumes". International Journal of

Computer Assisted Radiology and Surgery, 13(11), pp. 1741-1754. Springer.https:

//doi.org/10.1007/s11548-018-1831-6 (Chapter 9)

7. Tautz L, Walczak L, Georgii J, Jazaerli A, Vellguth K, Wamala I, Sündermann S, Falk V, and Hennemuth A (2019) "Combining position-based dynamics and gradient

https://doi.org/10.1007/s11548-019-02071-4 (Chapter 10)

8. Tautz L, Neugebauer M, Wamala I, Sündermann S, Falk V, and Hennemuth A (2018) "Automatic Detection of Commissures in Mitral Valve Geometry". Proceedings of

CURAC 2018, pp. 16–21. (Chapter 11)

9. Tautz L, Hüllebrand M, Steinmetz M, Voit D, Frahm J, and Hennemuth A (2017) "Exploration of Interventricular Septum Motion in Multi-Cycle Cardiac MRI". In:

Proceedings of the Eurographics Workshop on Visual Computing for Biology and Medicine, pp. 169-178. Eurographics Association.https://doi.org/10.2312/vcbm.

20171251 (Chapter 12)

In publications 1, 3, 5, 6, 7, 8, and 9, I (LT) was sole first author and main contrib-utor to implementation, experimental evaluation and paper writing. In publications 2 and 4, I (LT) was co-author and contributed one of the evaluated algorithms and to paper writing.

Additional ideas and figures have appeared previously in the following publica-tions:

i. Tautz L, Feng L, Otazo R, Hennemuth A, and Axel L (2016) "Cardiac function anal-ysis with cardiorespiratory-synchronized CMR". Journal of Cardiovascular

Mag-netic Resonance, 18(S1), p. W24.https://doi.org/10.1186/1532-429X-18-S1-W24

ii. Tautz L, Hennemuth A, Chitiboi T, and Kramer U (2014) "Evaluation of a phase-based motion tracking method for the calculation of myocardial stress and strain from tagged MRI". Journal of Cardiovascular Magnetic Resonance, 16(1), p. P365. https://doi.org/10.1186/1532-429X-16-S1-P365

The efforts leading up to this thesis have been accompanied by a number of individ-uals, without whom this work would not have been possible.

First and foremost I would like to thank most sincerely Prof. Dr.-Ing. Anja Hen-nemuth for introducing me to the fascinating and challenging world of cardiac prob-lems, for working with me on countless projects, publications and prototypes, for giv-ing frank and direct feedback on my writgiv-ing, for maintaingiv-ing a constant level of sweets in our office, for guiding and shaping me on my way to become a scientist, and last but not least for supervising this work and pushing me to get it done with.

Over the years, I had the chance to meet and work with many extraordinary and amazing people at Fraunhofer MEVIS (in Bremen, Lübeck and Berlin) and the Insti-tute for Imaging Science and Computational Modelling in Cardiovascular Medicine. I would like to thank in particular my former office colleagues Prof. Dr. rer. nat. Richard Rascher-Friesenhausen, Jumana Al Issawi, Andreas Weihusen, Dr. rer. nat. Sebastian Meier, Dr. Hanieh Mirzaee, Jennifer Nitsch and Nora Breutigam for being most enjoyable and exciting company in the face of work. Out of all the friends and colleagues at MEVIS, I would like to thank especially Dr. Ola Friman, Dr. rer. nat. Ste-fan Wirtz and Prof. Dr.-Ing. Horst Hahn for teaching me in my early years, Prof. Dr. Jan Modersitzki and Dr. Jan Rühaak for fruitful discussions about image registration and quadrature filters, Dr. Jan Moltz for arguments about statistics and travels to Nor-den, Dr. rer. nat. Joachim Georgii for invaluable and concise mentoring and advice whenever I needed it, and the people of the Cardio group, Dr. Hans Drexl, Markus Hüllebrand, Hanieh, Dr.-Ing. Mathias Neugebauer, Dr. rer. nat. Lars Walczak, Dr. Teodora Chitiboi and Lilli Kaufhold, for sharing your fascination for the heart, coding, weird historical anecdotes and amusement with me.

Since my work is embedded in a scientific and medical environment, of course I would not have been able to accomplish all this without help, counsel and expertise from my collaborators and my clinical partners. I’d like to mention most notably Prof. Dr. Hans Knutsson, who introduced me to the Morphon, Prof. Dr. Leon Axel for vivid insights into the inner workings of the heart, Dr. Franziska Seidel and PD Dr. Simon Sündermann for teaching me much-needed knowledge about all the cardiac details, and Dr. Isaac Wamala for giving advice, opinion or annotation regardless of the time. Beyond science and the workplace, I would like to thank Ruben Stein, Katharina Schumann, Christian Kanthak, Michaela Jesse, Olaf Klinghammer, Lars Bornemann, Sabrina Brockmann and Carsten Steinmetz for years of amazing and colorful stories, Dr. Christian Schumann, Dr. Christian Rieder, Christoph Brachmann, Dr.-Ing. Jan-Martin Kuhnigk, Dr. Florian Weiler and Dr. Jan Strehlow for enjoyable nights with mediocre movies and superb drinks, and Dr. Ingo Klöcker for rewarding climbing

ses-My deepest gratitude goes to my parents and my sisters for supporting me all along the way, and for always knowing that it would work out in the end.

My final and most special thanks go to my caring fiancée Steffi, who coped with me and my working hours especially in the final phase of my PhD, and gave me every bit of support I needed.

I The Heart in Motion 5

1 Introduction 7

1.1 Goals of This Thesis . . . 8

1.2 Contributions of This Thesis . . . 9

1.3 Outline . . . 11 2 Medical Background 13 2.1 Anatomy . . . 13 2.2 Physiology . . . 14 2.3 Diseases . . . 15 2.4 Imaging . . . 16 2.5 Image Processing . . . 19 3 Related Work 21 3.1 Analysis of Myocardial Wall Motion . . . 21

3.2 Mitral Valve Analysis . . . 23

3.3 Multi-Parameter Data Exploration . . . 25

3.4 Abbrevations . . . 28

II Analysis of Myocardial Wall Motion 31

4 Motion Analysis with Quadrature Filter Based Registration of Tagged MRI

Sequences 33

5 Benchmarking Framework for Myocardial Tracking and Deformation

Al-gorithms: An Open Access Database 45

6 Quadrature Filter Based Motion Analysis for 3D Ultrasound Sequences 63 7 3D Strain Assessment in Ultrasound (Straus): A Synthetic Comparison of

Five Tracking Methodologies 73

III Mitral Valve Analysis 97

9 Extraction of Open-State Mitral Valve Geometry From CT Volumes 99 10 Combining Position-Based Dynamics and Gradient Vector Flow for 4D

Mi-tral Valve Segmentation in TEE Sequences 115

11 Automatic Detection of Commissures in Mitral Valve Geometry 127

IV Concepts for Clinical Integration 135

12 Exploration of Interventricular Septum Motion in Multi-Cycle Cardiac MRI 137

V Summary and Conclusion 149

13 Concluding Discussion 151 13.1 Summary . . . 151 13.2 Discussion . . . 153 14 Outlook 157 Bibliography 159 2

2.1 Detailed sectional anatomy of the heart (Image from: Blausen.com staff (2014). "Medical gallery of Blausen Medical 2014". WikiJournal of Medicine 1 (2): 10. doi:10.15347/wjm/2014.010. ISSN 2002-4436). . . 14 2.2 Phases of the heart cycle (Image from: c User:Usien6 / Wikimedia

Com-mons / CC-BY-SA-4.0). . . 15 2.3 Contrast-enhanced cardiac CT in long-axis (upper row, left) and

short-axis view (upper row, right). Note the strong difference in intensity be-tween left and right ventricle caused by the injected contrast agent. Dif-ferent phases from diastole through systole to diastole (lower row). Im-ages courtesy of the Department of Cardiothoracic and Vascular Surgery, German Heart Center Berlin. . . 17 2.4 Cine MRI in long-axis four-chamber (upper row, left) and short-axis

view (upper row, right). Different phases from diastole through systole to diastole (lower row). Images courtesy of the Department of Cardio-thoracic and Vascular Surgery, German Heart Center Berlin. . . 18 2.5 TEE ultrasound in long-axis four-chamber (upper row, left) and

short-axis view (upper row, right). Note the different coverage of heart cham-bers when compared to the previous imaging modalities. Different phases from systole through diastole to systole (lower row). Images courtesy of the Department of Cardiothoracic and Vascular Surgery, German Heart Center Berlin. . . 19 2.6 Tagged MRI in long-axis four-chamber (left) and short-axis view (right).

Note the grid pattern in soft tissue, and the lack of pattern in fluids and air. Different phases from diastole through systole to diastole (lower row). Note how the pattern is fading over the heart cycle. Images cour-tesy of the Department of Cardiothoracic and Vascular Surgery, German Heart Center Berlin. . . 20

The Heart in Motion

Introduction

The human heart is the central driver of the circulatory system, which provides the body with oxygen and transports various nutrients and waste products. Heart func-tion is determined by the complex interacfunc-tion between active and passive structures in the left and right heart, blood pressure and flow conditions, and respiration.

Diseases of the heart and other parts of the circulation are called cardiovascu-lar diseases (CVD) and include myocardial infarction, stroke, heart failure, coronary artery disease, arrhythmia, congenital heart disease, and valvular heart disease. While the underlying mechanisms of these diseases vary, many are associated with lifestyle and aging. Other factors, such as genetic predisposition, congenital conditions, and injuries, also contribute to detrimental changes and disease progression.

According to the World Health Organization, the European Heart Network and other sources, cardiovascular diseases are the leading cause of death globally. With a prevalence of 88 million cases in Europa (data from 2015 [218]) and 121.5 million cases in the United States (data from 2016 [16]), CVDs present a major burden to pa-tients, society and national health care systems. In addition to lifestyle changes and medication for initial and mild cases, each year a large number of interventions and surgical procedures are performed to treat patients in advanced disease states. In Germany alone, in 2017 approximately 378,000 percutaneous coronary interventions (PCI) have been performed to treat the coronary arteries, as well as around 179,000 surgical procedures [15, 93]. The latter include around 34,000 heart valve procedures. In the United States, a total of 7,911,000 inpatient cardiovascular operations and pro-cedures were estimated for 2014 [16], including 480,000 PCIs and 156,000 heart valve procedures.

Medical imaging plays an important role in diagnosis, treatment planning and outcome monitoring. In recent years, a trend has developed to personalize image-based quantification and modeling to facilitate understanding of and tailor treatment to a specific phenotype and course of disease. Characterizing regional changes in morphology and function of the heart can indicate early stages of disease, as well as progression. In particular the local shape and dynamics of the heart walls and the four valves are considered good indicators of disease and remodeling. In valvular heart

their anatomy can be a valuable tool to analyze access ways, to assess risks in surgical strategy, and to plan complex repairs. Such a model can not only communicate the status quo and the specific mechanism of disease, but can also be used to try out sur-gical options in a virtual environment to pick the optimal strategy. As multiple struc-tures and physiological processes are interlinked in the cardiovascular system, the integration of image, sensor and clinical data on different levels enables a compre-hensive analysis and assessment of heart function. A visualization and exploration approach has to incorporate multiple temporal dimensions, such as the respiratory cycle, the contraction cycle and the variation across different heartbeats. For diagno-sis and decision making, it is highly relevant not only to know how specific structures move and change, but also when they do in relation to the underlying dynamic pro-cess.

1.1 Goals of This Thesis

Out of this trend for personalization and data integration in cardiac image analysis, the research community has been investigating a wide range of topics, including the aforementioned tasks. Based on these general directions, we formulate three goals that shall be addressed in this thesis:

Goal 1 Analysis of local wall motion to characterize changes for the assessment of

disease or regeneration

Goal 2 Characterization and quantification of mitral valve anatomy and dynamics to

support decision-making in therapy planning

Goal 3 Integrated exploration of multi-parametric cardiac data with different

abstrac-tion levels (patient, heart cycle, heart phase) and temporal dimensions (heart phase, respiration phase) to make analysis results accessible to clinical experts For Goal 1, we will investigate the tracking of wall motion in time-resolved cardiac image series acquired using tagged MRI and echocardiography to quantify and char-acterize regional motion. For Goal 2, the goal is a 4D segmentation of the mitral valve in clinically relevant CT and TEE imaging. The resulting 4D model can be utilized for motion and shape analysis, as well as for simulation of the valve dynamics. As for Goal 3, we will explore how image-based quantification results, such as from the tech-niques developed in the context of the first two goals, can be combined with patient-level data while providing the temporal context of heart phases and cycles necessary for interpretation. This multi-dimensional data exploration is an important step for the usability of quantification methods in a clinical setting.

This thesis combines publications that contribute to the given goals and have been published in peer-reviewed journals or presented at conferences with a comparable review standard. This section gives an overview of the publications, and how they relate to the stated goals.

1.2.1 Motion Analysis with Quadrature Filter Based Registration of Tagged MRI Sequences

Tagged MRI adds a magnetization pattern, typically lines or a grid, to an anatomi-cal acquisition, resulting in enhanced contrast within otherwise homogeneous struc-tures. It is considered a useful imaging protocol to assess myocardial deformation. De-saturation of the pattern causes the tagging grid to fade over time, which com-plicates analysis in affected phases. Modern CSPAMM MRI sequences compensate tag fading, but still exhibit intensity inhomogeneities. In earlier work, we applied quadrature filter to the problem of motion correction in cardiac first-pass perfusion imaging [181]. The tracking capability of quadrature filters is invariant to intensity changes, making the technique attractive for this type of image series. At the same time, they provide a dense motion field, exploiting the increased structural resolu-tion of the tagged MR images. In the 2011 STACOM Moresolu-tion Tracking challenge, we adapted the algorithm to work efficiently with 3D tagged MRI images. An iterative multi-resolution scheme allows to capture motion on different scales. As the convolu-tion required by the quadrature filters is computaconvolu-tionally expensive in 3D, we instead calculated motion independently in three orthogonal directions, and combined the results to a full 3D motion field. This contribution addresses Goal 1 by introducing an image-driven approach to track wall motion in tagged MRI, as well as in 3D ultra-sound.

1.2.2 Benchmarking Framework for Myocardial Tracking and Deformation Algorithms: An Open Access Database

Being the technical part of the STACOM 2011 Motion Tracking Challenge submission, Contribution I contained only a brief evaluation of the results. In Contribution II, the challenge organizers brought together all algorithmic contributions, and included an extensive evaluation of the submitted tracking results. Basis for the benchmarking efforts of motion tracking algorithms was a data set of 3D tagged MRI, cine MRI and ultrasound images, for which ground truth landmarks were available. Algorithm per-formance was measured in accuracy of landmark tracking, and by calculating strain curves to characterize wall motion. This contribution, too, addresses Goal 1 by pro-viding an objective performance evaluation of myocardial wall motion tracking in MRI and US with respect to reference data and other tracking approaches.

Contribution III builds upon Contribution I, where we adapted the multi-scale and regularization scheme to the image properties of simulated 3D echocardiograms pro-vided by the STACOM 2012 Motion Tracking Challenge. In particular, noise charac-teristics and large image extents required adaptations to the algorithm. This con-tribution addresses Goal 1 as well by proposing a refined image-driven approach to efficiently track wall motion in simulated 3D ultrasound.

1.2.4 3D Strain Assessment in Ultrasound (Straus): A Synthetic Comparison of Five Tracking Methodologies

Similar to Contributions I and II, Contribution III was the technical part of the STA-COM submission. Contribution IV contains a collation of all algorithmic submissions, together with a detailed evaluation based on ground truth segmentations and strain calculation.

1.2.5 Analysis of Cardiac Interventricular Septum Motion in Dierent Respiratory States

After a number of experiments and studies involving cardiac real-time MRI [33, 78, 196], we applied the well-tried motion tracking approach in a related setting. A novel MRI acquisition technique, XD-GRASP, allows to acquire multiple cardiac cycles while sorting image frames by cardiac and breathing phase. In a pilot study, we investigated how the motion of the interventricular septum (IVS) could be tracked and analyzed. This publication contributes a motion tracking approach for the IVS, thereby address-ing Goal 1, as well as initial experiments on the presentation of the multi-dimensional results, addressing Goal 3.

1.2.6 Extraction of Open-state Mitral Valve Geometry From CT Volumes

Analysis and quantification of mitral valve geometry is an important part of interven-tion planning. The delicate valve leaflets are difficult to extract from closed phases, and still challenging in open phases. To support quantification and model generation independently of image contrast and quality, we proposed a semi-automatic segmen-tation approach. An initial model is created from user-defined landmarks, and the model is subsequently fitted to the image data to exploit both expert knowledge in uncertain areas, and strong image information where available. This contribution addresses Goal 2 by providing a method to create an open-phase valve model.

Given Contribution VI and previous experience with motion tracking, the next obvi-ous step was combining the techniques into a 4D segmentation method which tracks a valve model over time. Early experiments indicated that the quadrature-filter-based approach could not cope with the rapid leaflet motion, and that a model enforcing topological consistency was required. By combining a simpler attracting force with a position-based dynamics model, we could formulate this as an iterative active sur-face problem. Contribution VII addresses Goal 3 by providing a method to create a 4D valve segmentation by tracking an initial open-phase valve model to the closed state.

1.2.8 Automatic Detection of Commissures in Mitral Valve Geometry

As assessment and repair of mitral valve defects make use of regional divisions, we proposed a method for automatic subdivision of a given valve model to support quan-tification using domain-specific reporting schemata. Together with Contributions VI and VII, Contribution VIII allows regional quantification of the valve in arbitrary phases, thereby addressing Goal 2.

1.2.9 Exploration of Interventricular Septum Motion in Multi-cycle Cardiac MRI

Following the preliminary work in Contribution V and other experiments with real-time MRI data, we studied a hierarchical visualization/exploration approach to ana-lyze cardiac multi-facet or multi-dimension data. By combining concepts from Coor-dinated Multiple Views and temporal synchronization, we enabled the exploration of parameters on different abstraction levels, while still providing the contextual infor-mation from heart and respiratory cycles.

1.3 Outline

The remainder of this thesis will be structured as follows. Chapter 2 gives a detailed in-troduction into the anatomy, physiology and imaging of the heart to provide medical context. Chapter 3 examines and summarizes existing work related to the three goals. Then, chapters 4 through 12 contain the individual contributions to this thesis, which have been previously published. Chapter 13 provides a summary and discussion of the presented contributions, and chapter 14 offers a perspective on future work.

Medical Background

In this chapter, an introduction into anatomy and physiology of the human heart is given, followed by an overview of clinical imaging and image-based analysis in this domain [102].

2.1 Anatomy

The heart consists of four chambers, the left and right ventricle and the left and right atrium, and is enclosed by the lung and by the organs of the abdomen. The venae cava transport de-oxygenated blood from the body through the right atrium to the right ventricle, from where it is pumped through the pulmonary artery to the lung. From there, oxygenated blood returns through the pulmonary veins and the left atrium to the left ventricle. The left ventricle then pumps blood through the aorta into the body by contraction of the heart muscle (myocardium). Heart muscle tissue is supplied by the coronary arteries, which branch from the aorta. The left and right ventricle are separated by the interventricular septum (IVS). Fig. 2.1 shows the anatomy of the heart.

The heart chambers are connected by four valves: the tricuspid valve between right atrium and right ventricle, the pulmonary valve between right ventricle and pul-monary artery, the mitral valve between left atrium and left ventricle, and the aortic valve between left ventricle and aorta. The valves prevent back flow during either ventricle filling (tricuspid and mitral valve) or ejection (pulmonary and aortic valve). The aortic, pulmonary and tricuspid valves consist of three cusps or leaflets, while the mitral valve has two leaflets. The chordae tendineae connect the tricuspid and the mitral valve leaflets to the papillary muscles, which are anchored in the right and left ventricle wall.

doi:10.15347/wjm/2014.010. ISSN 2002-4436).

2.2 Physiology

The heart cycle can be separated into distinct segments by electrical and mechan-ical events. Muscle contraction is triggered by electrmechan-ical currents that can be mea-sured using an electrocardiograph (ECG). In the electrocardiogram, three prominent waves are observable. The P wave marks atrial depolarization, the QRS complex re-sults from ventricular depolarization, and the trailing T wave is caused by ventricular repolarization. Initiated by the electrical changes, first the ventricles are filled, fol-lowed by atrial contraction (mid and late ventricular diastole). In ventricular systole, after isovolumetric contraction ventricular ejection follows, pumping blood into both circulations. In the subsequent early (ventricular) diastole, isovolumetric relaxation commences filling again. Fig. 2.2 shows the ECG and the heart in the different phases. While the heart contraction is a high-frequency process, breathing overlays the motion of the heart with a low-frequency pattern. Respiration is an alternation of inspiration and expiration, triggered by muscle contraction and pressure gradients. The distribution of blood between the pulmonary and the systemic circulation, as well as the position of the heart within the chest, is affected by the breathing state.

2.3 Diseases

Acquired heart disease is a major cause of death world-wide. The interplay of the complex anatomy with blood circulation and respiration make diagnosis and therapy planning in these diseases challenging. The left heart muscle plays a major role in heart function, and assessment and quantification of regional changes in contraction and tissue composition can support diagnosis and treatment selection. Likewise, the mitral valve is integral to left heart function, and because of its complex anatomy, quantification of the valve apparatus geometry and its dynamics can provide valuable information for therapy decision support.

The methods described and discussed in the following chapters are relevant to a number of cardiovascular diseases, including:

Myocardial infarction occurs when blood supply to the heart muscle is impaired or

blocked, causing the muscle tissue to die.

Heart failure manifests when the heart is unable to supply the body sufficiently,

typ-ically under physical stress or exercise. It can be caused by different structural or functional changes in the heart, such as infarction, high blood pressure, in-flammation of the heart muscle (myocarditis), damage to the heart muscle (car-diomyopathy) caused by infections or substance abuse, or congenital heart de-fects.

Arrhythmia is a family of conditions where the heart beat is too fast, too slow or

ir-regular. The rhythm variation is caused by disturbed electrical conduction. 15

atrium.

Mitral stenosis is another valvular heart disease where the mitral valve does not open

completely during diastole, impairing the filling of the left ventricle.

2.4 Imaging

Medical imaging is used to acquire structural and functional images of the human body. Commonly used imaging modalities comprise computed tomography (CT), magnetic resonance imaging (MRI) and ultrasound/echocardiography (US, echo). These modalities in general acquire images slice-wise, and depending on the imaging pa-rameters can image the target area as a stack of sparse slices or a fully covered vol-ume. The resolution in the acquisition plane is typically higher than the resolution in through-plane direction. Cardiac imaging and image analysis is an established com-ponent of clinical diagnostics and treatment planning [114, 52, 38].

2.4.1 Computed Tomography

The capability of the different imaging modalities to depict specific structures de-pends on the underlying mechanism of acquisition. CT uses 3D reconstruction of ray passage through the body, and displays images with quantitative values of X-ray attenuation (Fig. 2.3) [26]. It has very limited contrast within soft tissue, and the spatial and temporal resolution is dependent on radiation exposition. Metal implants, for example pacemaker wires, cause strong artifacts in the obtained images.

2.4.2 Magnetic Resonance Imaging

MRI uses magnetic fields and radio waves to non-invasively create images with dif-ferentiated contrast between tissue types and other materials [23]. While cine MRI sequences provide anatomical information (Fig. 2.4), tagged MRI superimposes spe-cial magnetization patterns on the image, increasing the spatial resolution of imaged structures (Fig. 2.6). In addition to anatomical imaging, MRI is used in first-pass per-fusion and late-enhancement imaging (both requiring contrast agent), tissue char-acterization via T1 and T2 mapping, and phase-contrast imaging for the depiction of blood flow. Ferromagnetic implants prohibit the use of MRI, while other types of implants can create image artifacts, such as dropout.

2.4.3 Echocardiography

Ultrasound uses the reflection of ultrasonic waves at tissue boundaries for imaging. It has a high spatial and temporal resolution, but is prone to artifacts related to wave

right ventricle caused by the injected contrast agent. Different phases from diastole through systole to diastole (lower row). Images courtesy of the Department of Cardio-thoracic and Vascular Surgery, German Heart Center Berlin.

row). Images courtesy of the Department of Cardiothoracic and Vascular Surgery, German Heart Center Berlin.

propagation. Signal dropout can cause structures to disappear partially, and shadow-ing can conceal structures located behind each other with respect to the transducer. Ultrasound imaging of the heart is called echocardiography [115, 80]. Echocardio-grams can be acquired with a non-invasive transthoracic (TTE) or an invasive trans-esophageal transducer (TEE) (Fig. 2.5). TEE is used in particular as intra-surgery plan-ning and monitoring modality. Echocardiography allows 2D and 3D acquisitions, as well as Doppler imaging to measure blood flow. Image quality is more strongly asso-ciated with operator skills and experience than in CT or MR imaging.

2.4.4 Time-resolved Imaging

All these modalities can be used to acquire time series. In cardiac imaging, the ac-quisition of a single image is often triggered by an ECG signal to obtain images from the same heart phase. If the acquisition speed is too low to capture all phases of a heart cycle, data from different cycles can be combined to form an averaged image series representing a single cycle. In newer approaches, faster acquisition times allow to capture multiple phases in 2D or 3D, providing true multi-cycle information. Un-less the heart is arrested for surgery, image frame are always captured from a dynamic process, introducing characteristic motion artifacts to all image series. Time series of the heart are the basis for motion analysis [190]. In some patients, ECG triggering is hampered or impossible due to factors such as arrhythmia or age, limiting the usable

pared to the previous imaging modalities. Different phases from systole through di-astole to systole (lower row). Images courtesy of the Department of Cardiothoracic and Vascular Surgery, German Heart Center Berlin.

modalities. Figs. 2.3, 2.4, 2.6 and 2.5 show a selection of heart phases from such time series.

2.5 Image Processing

In the past decades, medical image processing and analysis have evolved to an im-portant part of image-based interventions, surgery planning, and of course radiology [112]. Based on elemental computer vision techniques such as image segmentation and registration, applications such as quantification and volumetry, visualization, in-tervention planning, anatomical modeling, follow-up monitoring and motion analy-sis have been made possible [52, 38]. In the cardiac domain, in particular the model-ing of 3D anatomy [59] and 4D dynamics [164] have been investigated.

Exploiting the high spatial and temporal resolution and superior tissue contrast, research in MRI-based image analysis has produced diverse techniques [22], for ex-ample in quantification and modeling [14, 226], utilization of contrast-enhanced imag-ing [72], and motion analysis [162, 79, 81].

For image registration applications, cardiac images provide a number of chal-lenges. First of all cardiac motion is a superposition of different types of motion, including heart contraction, valve movement and breathing. Depending on the tem-poral sampling available to the modality and the depicted heart phases, image frames

phases from diastole through systole to diastole (lower row). Note how the pattern is fading over the heart cycle. Images courtesy of the Department of Cardiothoracic and Vascular Surgery, German Heart Center Berlin.

can show drastically different structures. For some structures with fast motion, such as the heart valves, the comparatively low sampling rate can cause blurring artifacts. To show all structures of interest in a time series with suitable resolution can also re-quire large image extents, imposing large computational efforts. For segmentation applications, the contrast between material or tissue types is crucial for image-based approaches. Information not visible in the images has to be estimated by prior or model knowledge. The spatial resolution limits the analysis of thin structures such as small vessels or heart valves. Again, fast-moving structures can appear blurred by motion.

Segmentation is commonly used in this image data to extract location and shape of structures of the interest such as the heart muscle and the heart chambers. Reg-istration applied to an image pair or images series provides motion information for the depicted structures, and allows fused visualization and analysis of complemen-tary image modalities. Exemplary applications are the analysis of heart muscle thick-ness, and the calculation of mechanical strain in the heart muscle. Such information is used to assess the dynamics and the regional mechanical properties of the heart muscle, which are indicators of functional muscle state.

Related Work

In this section, we will review work related to the three goals given in the introduction (Section 1.1). Section 3.1 covers the analysis of myocardial wall motion in cardiac MRI and echocardiography image data, Section 3.2 segmentation and analysis of the mitral valve in T and TEE data, and Section 3.3 the visualization and exploration of cardiac spatio-temporal multi-parameter data.

3.1 Analysis of Myocardial Wall Motion

Regional and local analysis of left myocardial wall motion has been a major research topic for many years. The commonly used imaging modalities for the visualization and quantification of wall motion are cardiac MRI and echocardiography, given their soft tissue contrast and high temporal and spatial resolution. Scott et al. presented a seminal overview of the types of motion affecting the cardiac system, and how they can be imaged [162]. Young discussed the role of MRI for the quantification of cardiac function and motion [226]. Loizou et al. gave an introduction into speckle tracking in echocardiography [60]. Brecht et al. presented an overview of ultrasound-based strain and strain analysis [74]. The recent review of Amzulescu et al. summarized the current imaging modalities, tracking techniques and validation methods for cardiac strain analysis [10].

A variety of MRI approaches, such as cine MRI, tagged MRI, DENSE, SENC and phase-contrast MRI, have been explored for the imaging of myocardial motion. Of these, only techniques based on cine MRI or tagged MRI image series have gained widespread use.

The reviews of Wang and Amini [209], Chitiboi and Axel [32] and Scatteia et al. [156] provide comprehensive overviews of the progression of research in this field over the years. The 2011 STACOM Motion Tracking Challenge gathered several state-of-the-art approaches for motion quantification and analysis in cine and tagged MRI [192]. Muser et al. presented clinical applications of feature-tracking-based strain analysis [111]. Mäkelä et al. gave an overview of cardiac image registration in general [98].

and techniques utilizing biomechanical models for motion recovery. Rahman et al. give an overview of current feature tracking approaches [144].

Feature-tracking-based methods comprise the tracking of endocardial and epicar-dial surfaces [9, 167, 104, 127, 168, 126, 31, 147, 231], the tracking of specific shapes or sparse points [116, 225], and more recent, commercially available feature tracking software [101, 39, 189, 24, 131, 161]. Registration-based approaches encompass di-verse techniques such as finite element models (FEMs) [200, 199, 201], information-theory-based registration [134], elastic matching and models [31, 135], free-form de-formations [133, 108, 109], 4D spatio-temporally consistent registration [176], Demons-based approaches [99, 105], and optical flow [221]. Model-Demons-based approaches include FEMs [166], incompressible deformable models [17, 18] and the electrophysiological, electromechanical and biomechanical components of the physiome model [163, 219, 48, 146, 220]. Francone et al. used manual measurements to quantify interventricular septum motion in anatomical MR images [57, 58].

Methods for tagged MRI motion tracking can be grouped into landmark-based methods with a sparse motion field, methods using deformable models and interpo-lation to obtain a dense motion field, methods extracting motion information directly from the image using Fourier analysis concepts (frequency or phase), and registration-based approaches.

Landmark-based techniques have been used early for motion analysis [89, 8, 155]. A variety of deformable models have been proposed for the analysis task [229, 128, 129, 86, 130, 37, 29, 211, 223, 191, 84], including finite element models [227, 230, 228, 69, 210], statistical material models [76], and spline models [143, 7, 77, 124, 177, 212, 195, 49, 50, 194, 30]. Fourier-based methods include the family of harmonic phase analysis (HARP) approaches [119, 120, 150, 153, 118, 151, 125, 1, 56, 2, 154, 97], sine-wave modeling [13], and Gabor-filter-based analysis [107, 142, 232]. Hor et al. recently compared HARP with the newer feature tracking technique [75]. The proposed registration-based approaches come from different parts of the field, such as optical flow methods [55, 141, 51, 197, 224], free-form deformations [145, 28, 45, 108], information-theory-based registration [134, 149, 121], wavelets [123, 122] and Demons-based approaches [99, 105]. Shi et al. presented an approach for the com-bined analysis of cine and tagged MRI [215].

Motion analysis in echocardiography typically makes use of Doppler tissue imag-ing, tracking the RF signal, or tracking the extracted B-mode envelope. The latter is also referred to as speckle tracking [106]. The 2011 and 2012 STACOM Motion Track-ing Challenges gathered several state-of-the-art approaches for motion quantification and analysis in echocardiography data [192, 43]. De Luca et al. surveyed large-scale tracking and motion analysis approaches in general B-mode ultrasound [46]. As is the case in MRI processing, a variety of approaches have been proposed for speckle tracking, including free-form deformations [44] and elastic image registration [73].

The majority of research has shifted away from specialized imaging protocols and post-processing algorithms towards conventional image registration approaches and in particular speckle and feature tracking algorithms. Own work in this field can be grouped with the latter approaches, and has been compared with other state-of-the-art approaches in two MICCAI motion tracking challenges.

3.2 Mitral Valve Analysis

In this section, we will assess work related to the 3D and 3D+t segmentation and track-ing of the mitral valve in CT and TEE image sequences.

3.2.1 3D Segmentation in CT

The extraction of mitral valve geometry, comprising the mitral annulus, the leaflets and the chordae tendineae, is a topic that has seen several proposed methods in recent years for many major imaging modalities, such as CT, micro-CT, 3D and 4D echocardiography and MR imaging. Votta et al. give an overview of recent patient-specific geometry extraction methods [205].

Grbi´c et al. have proposed several approaches based on multi-layered spatial or spatio-temporal shape models. These models consecutively fit coarse-to-fine shape information (bounding boxes, landmarks and surfaces) to the image data to segment all heart valves, including the mitral valve. They have validated their approaches on a variety of CT, micro-CT and echocardiography data [83, 68, 100, 66, 67]. These tech-niques in general require extensive training on a representative data set. Their vali-dation using CT data included pathologies such as valve regurgitation, stenosis and prolapse, but was limited to data where the mitral valve had good contrast.

There are commercial software packages available where it is possible to automat-ically segment the mitral annulus from CT data [21] and to manually segment the valve, also from CT data [209]. Researchers also addressed the extraction of high-resolution valve and chordae geometry from micro-CT for model validation [66, 193, 91, 92], and the application of simple segmentation methods for 3D printing purposes [64, 206, 62].

For echocardiography data, there is a wide range of published methods to segment the mitral annulus or the complete valve in 3D and 4D data. Votta et el. presented a semi-automatic method that includes manual definition of the annulus and leaflets in an end-diastolic phase, followed by tracking the geometry over time [204]. Pouch et al. proposed a semi-automatic method to segment a mid-systolic valve where a manual outline of the valve and coaptation zone is used to initialize a level-set-based active contour segmentation [139]. They later introduced a multi-atlas segmentation method where atlases of open and closed valves are registered to 4D ultrasound data, followed by the fitting of a medial leaflet model [138]. Schneider et al. segmented the

with 4D ultrasound by identifying the closed valve phase and tracking the annulus throughout the image sequence [159]. Graser et al. fitted an annulus shape model to 4D ultrasound data after manual pose initialization [65].

For MRI data, there have been some approaches using landmark-based geometry extraction in custom software. Wenk et al. segmented the closed valve using manual landmarks [214]. Stevanella et al. manually defined annulus and leaflet landmarks in end-diastole, and created the leaflet surfaces by spline interpolation [173].

3.2.2 3D+t Segmentation in TEE Ultrasound

There is a wide range of approaches to segment the open valve, closed valve or full im-age sequences from echocardiograms. Methods capable of 4D segmentation typically segment the time frames individually. Schneider et al. proposed a segmentation algo-rithm based on manual initialization, graph cuts and the max-flow algoalgo-rithm to detect the closed mitral valve [158]. The same group later presented an optical-flow-based semi-automatic approach incorporating topological and image information to detect the open-state valve [157]. Zhou et al. suggested an unsupervised, purely data-driven 4D segmentation approach based on matrix spectral analysis and outlier detection in a low-rank representation of the TTE image sequence, which was compared against manual expert segmentations [234]. Liu et al. published an improved version of this approach, but included only minimal evaluation [96].

The multi-layer shape model framework of Ionasec et al. fits a spatio-temporal mitral-aortic model to end-diastole and end-systole. The fitted model instances are then propagated to other phases using a learned motion prior. The approach requires an extensive amount of training data [83, 67]. Pouch et al. approached the problem using a combination of multi-atlas joint label fusion and deformable modeling for the segmentation of open and closed phases [138, 137], which they later extended to a full 4D segmentation incorporating the representation of transitional phases and temporal consistency [136].

Sotaquira et al. proposed the segmentation of the closed valve based on man-ual initialization, anatomical priors and optimal path search [171]. Engelhardt et al. employed an interactive scheme of image-based segmentation of selected phases, fol-lowed by smooth interpolation and user correction to achieve a 4D segmentation [54]. Costa et al. used convolutional neural networks (CNN) to acquire a 4D segmentation, but did not comment on their choice of the training data [40].

Instead of segmenting every time frame individually, several groups have pro-posed to track the valve over time. Burlina et al. suggested to start from an open valve and track the leaflets to the closed state. Starting from manual initialization, image-based active contours and an finite element model with specific material prop-erties mimic the valve dynamics [27]. Schneider et al. used an image-based approach to transfer open-state geometry to the closed state using optical flow-based tracking

Pedrosa et al. used TTE image sequences from the CETUS challenge to propagate ex-pert contours on end-diastolic and end-systolic phases to other phases using optical flow and active surfaces, where the coaptation line is estimated by intensity analysis in the mesh neighborhood. Their segmentation approach is multi-layered, with each layer providing context information for the next task [132].

There are also several publications dedicated to segmentation or tracking for spe-cific valve parts. Veronesi et al. used optical flow tracking to follow mitral and aortic annuli [202]. Schneider et al. introduced a 4D annulus segmentation with a valve state prediction [159]. De Veene et al. used non-rigid registration to track the annulus [47]. Soerensen et al. used line tracking to follow leaflet motion in Doppler M-mode sequences [169]. Xia et al. proposed a diastolic phase segmentation with dropout correction [222]. Sultan et al. transformed the image sequence to a virtual M-Mode to segment the anterior mitral leaflet using active contours [175].

3.2.3 Summary

There have been several semi-automatic and automatic segmentation approaches for the mitral valve in different imaging modalities. Currently, no solution enables the segmentation of the annulus and the leaflets in 3D CT volumes when limited image quality restricts the differentiability of valve and surrounding structures. A common technique in the literature to define the valve in echocardiography or MRI data with-out context information or with poor quality is to use user-defined landmarks, and to fit the valve surfaces to these landmarks and structural information derived from the image [158, 173]. Such interactive fitting is a reasonable trade-off between over-fitting to data with varying quality, and over-simplification of the complex mitral valve geometry.

Many published 3D+t approaches require a substantial amount of training data [83, 67, 138, 137, 136, 40]. The training data, as well as priors on anatomy and leaflet motion, often reflect valve dynamics as seen in high quality images with limited patholo-gies. Purely data-driven methods in literature offer only limited evaluation [234, 96]. Two of the tracking approaches incorporate material or mechanical properties [27, 160], while a third one relies on a hierarchical image-based scheme [132]. In clinical practice, varying pathologies and image quality require a very robust approach that incorporates image data and optionally user inputs, but can cope with missing infor-mation and is not limited by a model of specific valve conditions.

3.3 Multi-Parameter Data Exploration

The visualization and exploration of complex scientific and biomedical data has been studied extensively. Tang et al. discuss common design choices when designing a visualization architecture [178]. Kehrer et al. provide a comprehensive overview of

tion and exploration approaches for biological data [41]. Preim et al. give a survey of techniques used for the hierarchical analysis of perfusion data [140].

The temporal dimensions and multi-parameter nature of cardiac and other biomed-ical data require a careful selection of appropriate views. Aigner et al. give an overview of methods for the visualization and exploration of time-series data [3, 4]. A variety of approaches have been proposed for the exploration of multivariate data [90]. In the context of cardiac data, cylindrical projections and bull’s eye plots are often used to visualize regional information of the left ventricle [203, 165, 216]. Other techniques to associate spatial and temporal information include the perfusogram [25], time-trajectory maps [11], and color-coded plots and height fields [117, 140]. Chung et al. suggested sorted glyphs to present multi-dimensional parameters and their char-acteristics [36]. Wang et al. reviewed the literature for simulation ensemble visualiza-tions [208].

A common and flexible approach for the visual analysis of hierarchical scientific and medical data is the Coordinated Multiple Views (CMV) technique [148] (also known as Multiple and Coordinated Views). The appropriate tasks, layout and usage of CMV have been studied well [213, 152, 174]. This technique combines several linked views on the data showing different projections or abstractions, and has been used for the analysis of large scientific data sets [110], perfusion analysis [117], tissue classifica-tion and MRI data fusion [20], chewing moclassifica-tion analysis [87], analysis of cohort data [172, 94], geovisualization [53, 63], urban data analysis [217], simulation ensemble data [85, 103], neuroscience data [12] health awareness analysis in social media [42], scientific X-ray images [233], patient stratification and biomarker discovery [6], and MRI spectroscopy data [61].

A related concept is Focus+Context, which aims to present details about an object while providing contextual information at the same time [95, 19]. Parallel Coordinates appear commonly as an abstraction layer in a top-down visualization strategy [82, 70, 71], and have seen recent research to tackle the problem of large data sets [5].

3.3.1 Summary

Several fundamental concepts for the interactive exploration of multi-parameter/multi-faceted and spatio-temporal data have been proposed, as well as techniques to visu-alize different abstraction levels in a hierarchy of information. However, there is no solution that integrates the full hierarchy of data (from images to patient-level in-dices) and combines it with temporal synchronization. Own preliminary work has focused on the quantification of interventricular septum motion in different heart cycles [179, 180, 34] and proposed to use color-coded parameter maps and parallel coordinates for visualization, but has only dealt in a limited way with the question of how the data can be presented and explored. The visual exploration contribution of this work will focus in the composition and adaptation of these concepts for cardiac

The following abbreviations are used throughout this thesis and the included publi-cations:

1D One-dimensional 2D Two-dimensional

2D+t Two-dimensional time series 3D Three-dimensional

3D+t Three-dimensional time series 3DTAG 3D tagged MRI

3DUS 3D ultrasound 4D Four-dimensional

AABB Axis-aligned minimum bounding box AHA American Heart Association

BSA Body surface area

CETUS Challenge on Endocardial Three-dimensional Ultrasound Segmentation CNN Convolutional neural network

CRS Cardiorespiratory-synchronized

CSPAMM Complementary SPAtial Modulation of Magnetization CT Computed tomography

CTA Computed tomography angiography CVD Cardiovascular disease

DENSE Displacement Encoding with Stimulated Echoes DICOM Digital Imaging and Communications in Medicine ECG Electrocardiography

ED End-diastole ES End-systole

FE, FEM Finite element (method)

GPU Graphics processing unit GVF Gradient vector flow HARP Harmonic phase ITK Insight Toolkit

IVS Interventricular septum LA Left atrium

LAD Left anterior descending artery lAX, LAX Long-axis

LBBB Left bundle branch block LCX Left circumflex artery LV Left ventricle

LVBP Left ventricular blood pool LVM Left ventricular myocardium LVOT Left ventricular outflow tract MI Mitral insufficiency

MICCAI Medical Image Computing and Computer Assisted Intervention Society micro-CT X-ray microtomography

MR, MRI Magnetic resonance imaging MV Mitral valve

PBD Position-based dynamics PCA Principal component analysis PVA Polyvinyl alcohol

RA Right atrium

RCA Right coronary artery RV Right ventricle

SENC Strain-encoded SNR Signal-to-noise ratio

SPAMM SPAtial Modulation of Magnetization SSFP Steady-state free precession

STACOM Statistical Atlases and Computational Modeling of the Heart TDFFD Temporal diffeomorphic free-form deformation

TE Echo time

TEE Transesophageal echocardiography TR Repetition time

TTE Transthoracic echocardiography US Ultrasound

vmtk Vascular Modeling Toolkit

XPBD Extended position-based dynamics XD-GRASP eXtra-Dimensional GRASP

Analysis of Myocardial Wall Motion

Motion Analysis with Quadrature Filter

Based Registration of Tagged MRI

Sequences

Reprinted by permission from Springer Nature Customer Service Centre GmbH: Springer Nature. Statistical Atlases and Computational Models of the Heart. Imaging and Mod-elling Challenges. Motion Analysis with Quadrature Filter Based Registration of Tagged MRI Sequences. Tautz, L, Hennemuth, A and Peitgen, HO. c _{January 2012. [183]} The final published version is available online at:

https: // doi. org/ 10. 1007/ 978-3-642-28326-0_ 8

Context Within the Thesis

This chapter presents a quadrature-filter-based motion tracking algorithm for 3D tagged MRI and ultrasound images. The 2011 MICCAI workshop "Statistical Atlases and Com-putational Models of the Heart: Imaging and Modelling Challenges" (STACOM) orga-nized a Motion Tracking Challenge, aiming at evaluating the performance and repro-ducibility of state-of-the-art motion tracking algorithms against a multi-modal and public data set. The proposed method addressed several challenges present in 3D cardiac motion tracking at the time: tag fading in tagged MRI, inhomogeneous in-tensities over time in both tagged MRI and US, computational costs of processing 3D+t image data, and the presence of global and local deformations in the data. As this method was submitted together with motion tracking estimates on the challenge data, the publication contains only a brief qualitative evaluation of the observed mo-tion and computamo-tional performance. The presented method and study address Goal 1 and represent Contribution I of this thesis.

The author list includes Lennart Tautz, Anja Hennemuth and Heinz-Otto Peitgen. The author of this thesis was the first author of the publication. He conceived and imple-mented the method, wrote the majority of the article, and conducted the experimen-tal evaluation together with Anja Hennemuth. In addition, Anja Hennemuth sup-ported method development, paper writing and gave scientific advice. Heinz-Otto Peitgen gave scientific advice.

Based Registration of Tagged MRI Sequences

Lennart Tautz, Anja Hennemuth, and Heinz-Otto Peitgen

Fraunhofer MEVIS, Bremen, Germany

Abstract. Analysis of tagged MRI is a valuable tool for assessing re-gional myocardial function. One major obstacle for existing methods based on feature extraction and registration is the desaturation of the tagging grid over time. We propose a method based on quadrature filters that is invariant to changes in intensity, robust with respect to the grid geometry and provides a dense motion field that allows for the analysis of both global and local movements. A multi-scale and multi-resolution scheme is used to cover different scales of motion and to speed up reg-istration. The described method has been integrated into a prototypical application and applied to a phantom data set and 15 volunteer data sets provided by the STACOM’11. The automatic detection of the 4D motion field took about 130 minutes per MRI data set and about 90 minutes per US data set and resulted in plausible motion fields, which will be quantitatively assessed within the motion tracking challenge at MICCAI 2011.

Keywords: Tagged MRI, Morphon, Registration, Quadrature filter.

1

Introduction

MRI tagging is a non-invasive imaging method for the assessment of regional

myocardial motion, thus having the potential of being an important tool for the

clinical evaluation of cardiac dysfunction [7]. The method is based on the labeling

of image regions with saturation planes, whose deformation can then be tracked.

Different approaches have been proposed to determine a motion field from the

tagged MRI sequences, including segmentation-based methods [16], analysis of

the harmonic phase (HARP) [14,1,12,9], optical flow or related signal processing

concepts (e.g., Gabor filters) [15,3,5], and conventional registration approaches

[4,13]. Because the tagging grid desaturates over time in common tagging MRI

sequences, the contrast to the surrounding structures decreases, and the grid can

fade completely. This presents a major obstacle for existing methods. Modern

CSPAMM sequences do not suffer from tag fading, but exhibit a lower spatial

resolution and can still show intensity inhomogeneities. We propose an

intensity-invariant registration method driven with local phase information obtained by

quadrature filters that produces a dense deformation field.

O. Camara et al. (Eds.): STACOM 2011, LNCS 7085, pp. 78–87, 2012. c

2

Materials and Methods

2.1

Image Data

The STACOM’11 organizers provided a total of 16 tagged 4D MRI and 4D US

data sets, acquired from 15 healthy subjects and one dynamic physical phantom.

This phantom allows compression and rotation with a speed that enables the

simulation of a heart cycle through compression and relaxation within 1s [8].

MRI Data. The MRI data sets were acquired with a 3D CSPAMM sequence

that enables the generation of 4D volumes with tag planes in three orthogonal

orientations [10]. Data was acquired in short-axis orientation (see Fig. 1). The

volunteer data sets were obtained with a voxel size of 0.96 x 0.96 x 0.96 mm

and 20 to 38 time points. The phantom was scanned with a spatial resolution of

1.01 x 1.01 x 1.01 mm

over 23 time points.

US Data. The US data sets were acquired with a full-volume apical view (see

Fig 5). The volunteer datasets have voxel sizes between 0.66 x 0.66 x 0.58 mm

and 0.96 x 0.96 x 0.72 mm

_{and consist of 11 to 24 time frames whereas the}

phantom dataset was acquired with a voxel size of 1.35 x 1.16 x 0.96 mm

_over

19 time points.

The temporal resolution of the data sets was not included in the provided

image information.

2.2

Method

The described method is based on the Morphon algorithm introduced by

Knutsson et al. [6]. In previous work, this method has been successfully

ap-plied to MRI perfusion data, where the problem of contrast variation appears

due to the wash-in and wash-out of contrast agent [11].

Background. Phase-based image registration is based on the Fourier Shift

Theorem, which states that the Fourier transforms of a signal f (x) and a shifted

signal f (x

− d) are related via a phase factor F{f(x − d)} = e

_F{f(x)}.

For two signals f

(x) = f (x) and f

(x) = f (x

− d), we have a d proportional to

arg



_F{f

(x)

}F{f

(x)

}



, with

denoting the complex conjugate. By using the

local phase φ(x), derived from the complex analytical signal f

(x) = A(x)e

of f (x), the above approach can be used to estimate non-stationary shifts in 1D.

The analytic signal is in practice estimated by applying a quadrature filter, q(x),

ˆ

f

(x) = (f

∗ q)(x), which has a band-pass character that determines the scale of

the structures or shifts of interest.

To generalize the analytic signal, which inherently is a 1D construct, to images

of higher dimensions, a set of quadrature filters q

(x) with different orientations

ˆ

n

is applied. The generalized analytic signal in direction ˆ

n

for an image I(x)

is then obtained as I

(x) = (I

∗ q

)(x). Assume now that we have a deformed

image J(x) = I(x + d(x)), where d(x) is an unknown deformation field that we

wish to estimate. The displacement ˆ

d

(x) along the orientation ˆ

n

can then be

estimated by the local phase difference of the complex product

p

(x) = I

(x) J

(x).

(1)

Following the Shift Theorem, ˆ

d

(x) is proportional to arg



p

(x)



. For each

oriented quadrature filter q

_{(x), a displacement estimate is obtained. A}

confi-dence measure c

(x) can be associated with the estimate in each filter direction,

c

(x) =



_

p

(x)









1 + cos



arg



p

(x)



(2)

and the individual measures contribute to a combined confidence measure c(x)

formulated as

c(x) =



c

(x).

(3)

The rationale behind this confidence measure is that the magnitude of p

(x) is

large if there is a strong response of the filter q

(x) in both images, indicating

similar structures. If the phase of p

(x), that is, the phase difference between

I

(x) and J

(x), is large, the quadrature filter has likely picked up different

structures, which makes the estimate less certain. A first estimate of the complete

deformation field can be formulated by weighting the displacement estimates

with the associated confidence measures

d(x) =

i

c

(x)d

(x)ˆ

n

c

(x)

.

(4)

Biological tissue generally deforms smoothly, and a spatial regularization should

be applied to reflect this prior knowledge in the deformation field d(x). By

applying so-called normalized averaging, the confidence measure contributes to

the regularized deformation field,

d

(x) =

[d(x)c(x)]

_{∗ g(x; σ}

₎

c(x)

_{∗ g(x; σ}

₎

,

(5)

where the division is taken voxel-wise, and g is a Gaussian kernel. In the resulting

field uncertain displacements have been penalized, allowing the convergence to

a smooth field.

To estimate large deformations, the displacement estimation outlined above

must be implemented in a scale space, and it is also necessary to iterate the

estimation several times on each scale to refine the estimation. The deformation

estimates are accumulated in d

(x) as

d

(x)

← d

(x) +

c(x)

c

(x) + c(x)