The estimation of the inverse virtual depth z is performed for each pixel xR = [xR, yR]T in the raw (sensor) imageIM L(xR). As already mentioned, the depth observations are obtained starting from the shortest baseline and working up to the largest possible baseline. Based on each new observation, the inverse depth hypothesis ZM L(xR) of a raw image pixel xR is updated, thus becoming more reliable.
ZM L(xR)∼ N(z(xR), σ2z(xR)) (5.3) To reduce computational effort, the first step is to check for each baseline whether the pixel under consideration xR has a sufficiently high intensity gradient along the epipolar line, as defined in eq. (5.4).
|gI(xR)Tep| ≥TH (5.4)
Here gI(xR) represents the intensity gradient vector at the coordinates xR (eq. (5.5)) and TH
represents a predefined threshold.
gI(xR) =gI(xR, yR) =h
∂I(xR,yR)
∂xR
∂I(xR,yR)
∂yR
iT
(5.5) Figure5.2bshows an example of the matching cost (cf. eq. (5.8)) dependent on the inverse virtual depth z of a certain image point for three different epipolar line angles. In this specific example the intensity gradient gI(xR) is almost orthogonal to e(1)p = [1,0]T. Therefore, no minimum is obtained in the cost function and hence, depth estimation does not have to be performed for this baseline. However, for the same point xR both e(2)p =
0.5,−√ 0.75T
and e(3)p = 0.5,√
0.75T
result in unique minima.
5.3.1 Stereo Matching
To find the pixel in a certain micro image which corresponds to the pixel of interest xR, it is searched for the minimum intensity difference along the epipolar line in the corresponding micro image.
If no inverse virtual depth observation for the pixel of interestxR has been obtained yet, an exhaustive search along the epipolar line must be performed. For this case, the search range is limited by the micro lens border on the one end and by the coordinates of xR with respect to the micro lens center on the other end. A pixel on the micro lens border results in the maximum observable disparityµ and thus in the minimum observable virtual depth v, while a pixel at the same coordinates as the pixel of interest in the corresponding micro image equals a disparity µ = 0 and thus also equals a virtual depth v = ∞. Furthermore, the virtual depth range is
5.3. VIRTUAL DEPTH ESTIMATION 51
lower bounded by the total covering plane (TCP). The total covering plane (TCP) defines the plane closest to theMLAfor which it can be guaranteed that any virtual image pointxV on the plane appears focused in at least one micro image. For a multi-focus plenoptic camera with a hexagonally arranged MLA, consisting of three different types of micro lenses, the TCP is at a virtual depth v= 2 (seePerwaß and Wietzke [2012]).
If an inverse virtual depth hypothesisZM L(xR) for a pointxRalready exists, the search range can be limited to z(xR)±nσz(xR), where n is usually chosen to be n= 2. In the following we define the search range along the epipolar line as given in eq. (5.6).
xsR(µ) =xsR0−µ·ep. (5.6)
Here xsR0 is defined as the coordinate of a point on the epipolar line at the disparity µ = 0, as given in eq. (5.7).
xsR0=xR+∆cM L·ep (5.7)
Within the search range, the sum of squared intensity differences eISS over a 1D pixel patch (1×N) along the epipolar line is calculated, as defined in eq. (5.8).
eISS(µ) =
N−1
X2
k=−N−12
[IM L(xR+kep)−IM L(xsR(µ) +kep)]2 (5.8)
The resulting estimate is the disparity µ which minimizes eISS(µ). For the implementation we found N = 5 to be a good choice for the patch size. It is important to emphasize that µ is estimated with sub-pixel accuracy by interpolating linearly between the samples of the intensity imageIM L(xR) and therefore the disparity is not restricted to any regular grid. In the following it is referred to the estimated disparity by ˆµ, which defines the corresponding pixel coordinate xsR(ˆµ).
Observation Uncertainty
For each inverse virtual depth observation, an observation uncertainty, represented by the inverse virtual depth variance σz2, is calculated. Hence, the variance σ2z defines a measure of certainty with which an estimate represents the corresponding real value. For the determination of the variance σz2, on the one hand, a photometric component is considered, which models the effect of sensor noise, and on the other hand a focus component is considered, which models the effect of differently focused micro images, as they occur in a multi-focus plenoptic camera (Perwaß and Wietzke[2012]).
Photometric Disparity Error The effect of sensor noise ǫn on the estimated disparity µ is modeled in a photometric disparity error, defined by the variance σµ(I)2 . This error source is inspired by Engel et al.[2013] and is modeled in a similar way.
The variance of the sensor noise σ2n can be considered to be the same for each pixel xR. In doing so, the effect of vignetting correction is neglected. To compensate for vignetting of the micro lenses pixel intensities at micro image boundaries and, accordingly, the additive noise components are amplified.
In the following it will be derived how σ2n affects the disparity estimation. To do this, we formulate the stereo matching by the minimization problem given in eq. (5.9), where the estimated
disparity ˆµis the one which minimizes the squared intensity difference eI(µ)2. In the interest of temporarily simplifying the expressions, the sum over a number of pixels is omitted, as defined foreISS(µ) in eq. (5.8).
ˆ
µ= arg min
µ
eI(µ)2
= arg min
µ
h(IM L(xR)−IM L(xsR(µ)))2i
(5.9) To find the minimum, the first derivative of eI(µ)2 with respect to µ is calculated and is set to zero. This results in eq. (5.11) as long asgI(µ)6= 0 holds true (which is guaranteed by eq. (5.4)).
∂eI(µ)2
∂µ = ∂[IM L(xR)−IM L(xsR(µ))]2
∂µ
= 2 [IM L(xR)−IM L(xsR(µ))]·[−gI(µ)] (5.10) 0=! IM L(xR)−IM L(xsR(µ)) (5.11) In eq. (5.10) the intensity gradient along the epipolar linegI(µ) is defined as follows:
gI(µ) =gI(xsR(µ)) = ∂IM L(xsR0−µep)
∂µ =gI(xsR(µ))Tep. (5.12) After approximating eq. (5.11) by its first-order Taylor-series at the disparity µ = µ0, which is close to ˆµ, it can be solved for ˆµas given in eq. (5.13).
ˆ
µ= IM L(xR)−IM L(xsR(µ0))
gI xsR(µ0) +µ0 (5.13)
If one now considers IM L(xR) in eq. (5.13) to be disturbed byAWGN, the variance σµ(I)2 of the disparityµ can be derived as given in eq. (5.14).
σµ(I)2 = Var{IM L(xR)}+ Var{IM L(xsR(µ0))} gI(xsR(µ0))2
= 2σn2
gI(xsR(µ0))2 (5.14)
Figure5.3illustrates how the gradientgI affects the estimation ofµ. The blue line represents the tangent at the disparityµ0, at which the intensity values are projected onto the disparities.
Focus Disparity Error In contrast to regular cameras, which generally are focused to infinity, the micro images of the plenoptic camera can not be considered to be in focus for the entire operating range, especially not for a multi-focus plenoptic camera (Perwaß and Wietzke [2012]).
Hence, the focusing itself also affects the stereo observation. In the following the varianceσ2µ(v,k,j) of the focus disparity error is derived.
Letkbe the index of the micro image for which mapping is performed, whilej is the index of the stereo reference. To model the focus disparity error, it is considered that the real edgeIreal(x), which is observed in the micro images, is a perfect Heaviside-step-function h(x) with amplitude A and offsetB along the epipolar line:
Ireal(x) =A·h(x) +B. (5.15)
5.3. VIRTUAL DEPTH ESTIMATION 53
gI(µ0)µ IM L(µ)
µ
gI(µ0)µ IM L(µ)
µ
Figure 5.3: Visualization of the photometric disparity error. Sensor noise ǫn can be considered additive white Gaussian noise (AWGN) which disturbs the intensity values IM L(xR) and thus affects the disparity observation asAWGN. As shown on the left, for a low image gradient along the epipolar line, the influence of the sensor noisenI is stronger than for a high image gradient, as it is shown on the right.
The variable x is the position on the respective epipolar line in relation to the step position µ0i (i∈ {k, j}):
x=µk−µ0k=µj −µ0j. (5.16)
Therefore, the correct disparity is defined by µ∗=µ0j−µ0k. During the imaging process the edgeIreal(x) is filtered by a Gaussian filter with varianceσ2i, which in turn depends on the virtual depthv, the micro lens type and the aperture of a pixel1.
Thus, on the sensor one receives the following intensity functions along the epipolar line:
IM Li(x) =B+A 2
1 + erf
x σi√
2
i∈ {k, j}. (5.17)
The estimated disparity ˆµ=µ∗+ǫf is the one for which both intensities have the same value and therefore, the following condition is fulfilled:
IM Lk(x)=! IM Lj(x−ǫf), (5.18) erf
x σk
√2 !
= erf x−ǫf σj
√2
!
. (5.19)
Since the error function (erf(·)) can not be solved analytically, eq. (5.19) is linearized and one obtains, after some rearranging, the following relationship between the focus disparity error ǫf
and the position on the edgex:
ǫf =x·
1− σj σk
. (5.20)
Figure5.4visualizes the focus disparity errorǫf, by way of example, for two different positions (x1 and x2), in relation to the real edge position on the epipolar line. Here, the red and blue curves represent the intensity along the epipolar line in two different micro images with different blur radii. For a certain positionxthe estimated disparity ˆµwill be the one for which both curves have the same value and therefore will be shifted by ǫf with respect to the real disparity µ∗, as
1Pixel aperture results rather in a sinc response function. However, the Gaussian kernel gives a good approxi-mation as defocus blur will generally be dominate.
x1 0 x2 B
A+B
ǫf1
ǫf2
x IML(x)
IM Lk(x) IM Lj(x)
Figure 5.4: Visualization of the focus disparity error ǫf for two different positions (x1 and x2), in relation to the real edge position, on the epipolar line. The red and blue curves represent the intensity along the epipolar line in two different micro images with different blur radii. For a certain position x on the edge the estimated disparity ˆµ will be the one for which both curves have the same value and therefore will be shifted by ǫf with respect to the real disparity µ∗. given in eq. (5.18). For the case that both images have the same blur radius, both curves are perfectly overlaid and therefore ǫf = 0 will hold for any position x.
Considering the position in relation to the real edgexas a random variable with variance σx2, the varianceσ2µ(v,k,j) of the focus disparity error is as follows:
σµ(v,k,j)2 =σx2·
1−σj σk
2
(5.21) The standard deviationσi (i∈ {k, j}) depends on the blur diametersi of the respective micro image, which is calculated based on the virtual depthv and the micro lens parameters (DM,fM, B) using the thin lens equation:
si =DM ·
1 vi
+ B
fM i −1
(5.22)
Since the minimum blur radius is limited by the pixel pitch and the overall optical system, the blur radius has a lower boundarys0. Thus the following variancesσ2k and σj2 result to be:
σk2=β2·min{s2k, s20}, σ2j =β2·min{s2j, s20} (5.23) The constant parameterβ models the scaling from blur diameter to the standard deviation of the Gaussian filter.
Considering the two error sources, photometric disparity error and focus disparity error, as independent random variables, the complete observation uncertainty for the inverse virtual depth z can be defined as given in eq. (5.24)
σ2z =∆c−2M L·
σ2µ(I)+σ2µ(v,k,j)
(5.24)