Probabilistic Morphable Models
Thomas Vetter
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Photo by Pete Souza/The White House via Getty Images.
Male 60-70 Blue eyes Wide nose Mouth closed
β¦β¦
βRobert Gatesβ
Analysis by Synthesis
3D
World Image
Analysis
Synthesis
Image Model Image Description
model parameter
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Example based image modeling of faces
2D Image 2D Face Examples 2D Image 3D Face Scans
= w
1 * +w
2 * +w
3 * +w
4 * +. . .Morphable Models for Image Registration
Output R = Rendering Function
Ο = Parameters for Pose, Illumination, ...
Optimization Problem: Find optimal Ξ± , Ξ², Ο !
R
ο²ο¦ ο§
ο½ ο§
ο§ ο§ο§
ο¨
οΆ ο·
ο· ο·
ο·ο· οΈ
Ξ² 1 + Ξ² 2 + Ξ² 3 + β― Ξ± 1 + Ξ± 2 + Ξ± 3 + β―
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Probabilistic Morphable Models
1. Model-based image registration using Gaussian Processes for shape deformations
2. βProbabilistic registrationβ: Find the distribution of possible transformations h(π) that transforms πΌ π to πΌ π .
?
π( π |πΌ π , πΌ π )
Gaussian Process Morphable Models:
ο΄ A Gaussian process β ~ πΊπ π, π on π is completely defined by its mean function
π βΆ π β β 3 and covariance function
π βΆ π Γ π β β 3Γ3
ο΄ A low rank approximation can by computed using the NystrΓΆm approximation.
β π β π + Ο π π π π π π Ξ¦ π with π ~ π(0, πΌ π )
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Advantage of Gaussian Process Morphable Models
ο΄ Probabilistic formalism !
ο΄ Extremely flexible concept. By varying the covariance function k a variety of βdifferentβ algorithms of deformation modelling are included.
ο Thin Plate Splines
ο Free Form deformations
ο β¦
ο Standard PCA-Model
βScalismoβ an open source library by Marcel LΓΌthi see also our MOOC on FutureLearn βStatistical Shape Modllingβ
Surface Data Prediction as Gaussian Process Regression
3D Surface Data Base
Analysis
3D Input Statistical
Prediction Original
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Surface Data Prediction
as Gaussian Process Regression
Application
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Disclocation of the patella
Femur Patella MRI-Slice
Example use-case: Trochlea dysplasia
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Trochlea-Dysplasia
Surgical intervention: Increase goove
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Surgical intervention: Augment bony structure
Posterior Shape Models
T. Albrecht, M. LΓΌthi, T. Gerig, T. Vetter, Medical Image Analysis, 2013
Automatic inference of pathology
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Probabilistic Inference for Image Registration
ο΄ Generative image explanation: How to find π explaining I ?
π π πΌ = β(π; πΌ) π(π)
π(πΌ) π πΌ = ΰΆ± β(π; πΌ)π(π)dπ ---> Normalization intractable in our setting
ο΄ What can be done:
1. Accept MAP as the only option
2. Approximate posterior distribution (e.g. use sampling methods)
The Metropolis-Hastings Algorithm
ο΄ Need a distribution which can generate samples: π π β² π)
ο΄ Algorithm transforms samples from π into samples from π:
1. Draw a sample π
β²from π π
β²π)
2. Accept π
β²as new state π with probability π
ππππππ‘= min
π πβ²π π π π|πβ² π πβ²|π
, 1 3. State π is current sample, repeat for next sample
---> Generates unbiased but correlated samples from π
ο΄ Markov Chain Monte Carlo Sampling: Result: π 1 , π 2 , π 3 , β¦ β¦
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
MH Inference of the 3DMM
ο΄ Target distribution is our βposteriorβ:
ο΄ π: ΰ·¨ π π πΌ
π= β π|πΌ
π, πΌ
ππ π
ο΄ Unnormalized
ο΄ Point-wise evaluation only
ο΄ Parameters
ο΄ Shape: 50 β 200, low-rank parameterized GP shape model
ο΄ Color: 50 β 200, low-rank parameterized GP color model
ο΄ Pose/Camera: 9 parameters, pin-hole camera model
ο΄ Illumination: 9*3 Spherical Harmonics for illumination/reflectance
ο΄ β 300 dimensions (!!)
Metropolis Filtering
ΞΈ
β²MH-Filter:
Q ΞΈ β² |ΞΈ
πππππππ‘reject ΞΈπππβ ΞΈβ²
ΞΈ
β²ΞΈ
β²update
ΞΈ
β²β ΞΈ
ο΄ Markov Chain Monte Carlo Sampling: Result: π 1 , π 2 , π 3 , β¦ β¦
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Results: 2D Landmarks
ο΄ Landmarks posterior:
Manual labelling: π
LM= 4pix Image: 512x512
ο΄ Certainty of pose fit?
ο΄ Influence of ear points?
ο΄ Frontal better than side-view?
Yaw, Ο
ππ= 4pix with ears w/o ears
Frontal 1.4
βΒ± π. π
ββ0.8
βΒ± π. π
βSide view 24.8
βΒ± π. π
β25.2
βΒ± π. π
βIntegration of Bottom-Up
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Metropolis Filtering
ΞΈ
β²MH-Filter: Prior
Q ΞΈ β² |ΞΈ
πππππππ‘reject ΞΈπππβ ΞΈβ²
update
ΞΈ
β²β ΞΈ
MH-Filter: Face Box
πππππππ‘
reject ΞΈπππβ ΞΈβ²
MH-Filter: Image
πππππππ‘
reject ΞΈπππβ ΞΈβ²
ΞΈ
β²π
0π
π π,πΉπ΅π π|πΉπ΅
π π,πΌπ π|πΉπ΅, πΌ
35
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Face analysis
Roger F.
asian caucasian blue eyes brown eyes wide nose male mustache gaze Hor yaw pitch roll
0.34 0.52 0.19 0.69 0.70 0.52 0.13 20Β°
34Β°
-8Β°
4Β°
Occlusion-aware 3D Morphable Face Models
Bernhard Egger, Sandro SchΓΆnborn, Andreas Schneider, Adam
Kortylewski, Andreas Morel-Forster, Clemens Blumer and Thomas Vetter International Journal of Computer Vision, 2018
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Face Image Analysis under Occlusion
41
Source: AFLW Database Source: AR Face Database
β π; πΌ = ΰ·
πππ₯ππ
β π; πΌ π₯
There is nothing like: no background model
βBackground Modeling for Generative Image Modelsβ
Sandro SchΓΆnborn, Bernhard Egger, Andreas Forster, and Thomas Vetter Computer Vision and Image Understanding, Vol 113, 2015.
= ΰ·
π₯βπΉπ
β π; πΌ π₯ Γ ΰ·
π₯βπ΅π
β π; πΌ π₯ β π; πΌ = ΰ·
π₯ β πΌ
β π; πΌ π₯
Maximum Likelihood Formulation:
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Occlusion-aware Model
π π; απΌ, π§ = ΰ·
π
π ππππ π; ΰ·© πΌ π π§ β π πππβππππ π; ΰ·© πΌ π 1βπ§
Inference
> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Initialisation: Robust Illumination Estimation
Initππππβπ‘ Initπ§
Initπππππππ
Results: Qualitative
Source: AR Face Database> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE
Results: Qualitative
49
Source: AFLW Database
Results: Applications
Source: LFW Database> DEPARTMENT OF MATHEMATICS AND COMPUTER SCIENCE