3.2 Optimization Procedure utilizing a Block Coordinate Ascent
3.2.3 Challenges Concerning the Convergence
Both the Iterations 3.2.2 and 3.2.1 demand very restrictive requirements in order to guarantee conver-gence. A solution vector x∗ which is an optimum in one subspace is not necessarily an optimum in an affine translated subspace, let alone on the global problem. In the context of camera placement, the optimum of the volume of the first camera’s coverage is not necessarily an optimum anymore if a sec-ond camera has moved, nor does it have to be an optimum of the volume of the fused coverage. These requirements and their challenges in camera network optimization are discussed in this section.
In order to see the difference of these optima, consider that for a maximizerx∗of an objective function f of the global problem (3.1) the following holds:
0∈argmax
x∈D
f(x+x∗)
In local optimization this corresponds to the gradient being zero at x = 0. An optimization procedure should terminate if we can assure this condition. Again,x+x∗might not be inDeven ifx∗∈ D, however again, we prefer to write the simplified subscript x∈ Dinstead ofx ∈ D −x∗. Similarly, one is able to define the termination of the subspace maximizations with the following definition.
Definition 3.2.5
Let f :D → Rbe a real-valued function and letV1× · · · ×VMbe a decomposition ofD. Additionally let the solutionx∗∈ Dbe given.
1. x∗is calledVm-subspace maximumof the subspaceVm,m∈ {1, . . . ,M}, if 0∈argmax
vm∈Vm
f(Umvm+x∗) (3.13)
2. x∗is calledstationary pointif it is aVm-subspace maximum for all the subspacesm=1, ...,M.
3. In contrast to the stationary point, we callx∗aglobal maximumif0∈argmax
x∈D
f(x+x∗).
In the upcoming paragraph, only one iteration step of 3.2.1 or 3.2.2 is discussed: We will see that there are functions for which aVm-subspace maximumx∗stays aVm-subspace maximum even if a coordinate ofx∗is changed which does not belong to the same subspace. This is necessary for the convergence of the BCA to a stationary point in one iteration step. However, stationary points are not necessarily global maxima. This is shown in the paragraphs thereafter. Since the objective function discussed in this thesis is particularly difficult for a BCA, these challenges have to be dealt with in the rest of this chapter.
Additive Separability
Figure 3.4 suggests that the achievement of the new solution x(i+1)is the same for the parallel as well as the sequential version. But this is only true for objective functions that areseparable, which means f can be decomposed intoMfunctionsφmthat each depend on a single subspace coordinate:
Definition 3.2.6
A function f : D → R is called additively separable on the decomposition (as in Definition 2.4.4) V1× · · · ×VMifMfunctionsφm:Vm→R,m=1, ...,M, exist with
f(x)=
M
X
m=1
φm(xm), where the subspace coordinates are given byx=[x1, . . . ,xM]∈ D.
The following corollary connects the notation of a separable function and a subspace maximum:
3.2. OPTIMIZATION PROCEDURE UTILIZING A BLOCK COORDINATE ASCENT 93
Corollary 3.2.7
For an additively separable function f the following holds: If x∗is a Vm-subspace maximum, the solution vector y = [x1, . . . ,xm−1,x∗m,xm+1, . . . ,xM]T is a Vm-subspace maximum for all subspace coordinates
With the property of additively separability the maximum of a function on a subspaceVmstays a maxi-mum when adjusting one of the other subspace coordinates of the other subspacesm=1, . . . ,m−1,m+ 1, . . . ,M. With this information, the Algorithm 3.2.1 converges within one iteration step to a stationary point if the subspace maximizations converge. Also, the procedures of Algorithm 3.2.1 and 3.2.2 come to the same result if the objective function is additively separable.
Lemma 3.2.8
Let f be an additively separable function with a single global maximum. Let x(i) ∈ D.
1. The result x(i+1) of an iteration step in both iterations 3.2.1 and 3.2.2 starting at the initial solution x(i)is the same. 2. x(i+1)is a stationary point. 3. x(i+1)is a global maximum.
Proof. 1. Let the solution vectorx(i+1)be the result of an iteration step of Iteration 3.2.2:
x(i+1)= Corollary 3.2.7 states that an addition of a term Pm
m=1Umum to the argument of the objective function f in argmaxvm∈Vm f(Umvm+ x(i)) does not change the subspace optimum, sincem < m.
This is the solution of one iteration step of Iteration 3.2.1.
2. Furthermore, the solution from Iteration 3.2.2 of a separable function of the form f(x) = PM
The previous lemma shows that both Iterations come to the same result for additively separable functions.
This result is a global maximum. Unfortunately, in camera placement the objective is not necessarily additively separable:
Example 3.2.4 (again):
After having found the best place for camera one, an equally designed second camera would indepen-dently be placed at the same spot. But this is not the best place for the second camera taking the placement of the first camera into account, as their fused coverage is larger if they are further apart. Here, the addi-tive separability depends on the possible positions of the camerasPand the geometry of the surveillance zone A: The fused coverage of two cameras placed in two separate rooms is indeed separable in the above sense.
With this example we have seen that the volume of the k-reliable coverage may be separable, but in important cases is not. Thus, the BCA will possibly not converge to a maximum in one iteration step.
Nevertheless, we have seen in Section 3.2.2 that a BCA yields some advantages for camera placement, so, the BCA needs to be further investigated as an iteration with more than one iteration step.
Stationary Points are not necessarily Global Maxima
Using additively separable functions, an obtained stationary point is always a global maximum. The objective function of camera placement is not necessarily additively separable. In this section, we reveal the issue of obtaining a stationary point in the interior of the domainDthat is not a global maximum.
Let us motivate the problem at hand, first.
Example 3.2.9
In order to illustrate the problem of Iteration 3.2.1 consider two functions that are slowly linearly ascend-ing on the diagonal from (−10,−10) to (10,10). One has a parabolic shape in the orthogonal direction and one is piecewise linear, as can be seen in the first plots of Figure 3.5 and 3.6.
Figure 3.5: Left: A differentiable function that is slowly, linearly ascending from (−10,−10) to (10,10) (left, red). The maximum is illustrated by a blue star. Right: Optimizing this function with Algorithm 3.2.1. Note that the graph is rotated by 90 degrees. The intermediate solution vectors (red crosses) are slowly oscillating all the way up to the maximum (blue star)
3.2. OPTIMIZATION PROCEDURE UTILIZING A BLOCK COORDINATE ASCENT 95
In the second pictures of these figures the intermediate and result solution vectors of the BCA are il-lustrated. While the BCA is slowly oscillating its way up to the top of the objective function in the differentiable case, the BCA in the second case is stuck in the same orthogonal subspaces over and over again. The reason: In the second example, the procedure has found a stationary point in the intersection of these subspaces, but this one obviously differs from the global maximum.
Figure 3.6: Left: A non-differentiable function that is slowly, linearly ascending from (−10,−10) to (10,10) (left, red). The maximum is illustrated by a blue star. Right: Optimizing this function with Algorithm 3.2.1. The intermediate solution vectors (red crosses) stay in the same subspaces after the BCA has found a stationary point which is not a global maximum (blue star).
The question is now whether there are any differentiable, stationary points which are not global maxima.
The optimality of a solution at a differentiable point of the objective function f is indicated by a vanishing gradient of the function. Generally, the gradient of a differentiable function at an interior point of the domain is zero if and only if the point is a local optimum of the function or a saddle point. We would like to be able to have a corresponding statement for the subspaceVmwith a suitable gradient:
Definition 3.2.10
Let (V1, . . . ,VM) be a decomposition ofDfor a function f which is differentiable at a pointx∗∈ D.
1. TheVm-subspace gradientof f is defined as∇mf(x∗) :=UTm∇f(x∗).
2. In contrast to the subspace gradient we call∇f theglobal gradientof f.
The problem illustrated in the Figures 3.6 and 3.5 is a result of the non-differentiabilities: If f was differentiable at the point of interest then either all the subspace gradients are 0 or no local optimum of the global problem has been found. This result is proven with the following lemma.
Lemma 3.2.11
Let f be a function which is differentiable at point x∗∈ iD, idenoting the interior ofD. Then
∇mf(x∗)=0for all subspaces m=1, . . . ,M ⇔ ∇f(x∗)=0
Proof. With the partition of the identity-matrix1n := (U1, ...,UM) ∈ Rn×n from Equation (3.12), the
which shows the claimed equivalence at differentiable points of the domain.
Therefore, if a function is differentiable at a solution vector x the stationary point corresponds to a local optimum or a saddle point. If a direction exists in which the solution can be improved, the sta-tionary point is not a saddle point. Thus, the issue of Figure 3.6 in Example 3.2.9 results from the non-differentiabilities, since the direction (1,1) improves the solution.
Now, one could argue that the non-differentiabilities of the objective f are so rare that this problem will not cause harm. However, in the concrete case of the placement of several cameras this issue does occur, as the following example shows. Thus, we need to figure out a way to smoothen the objective function when dealing with non-differentiable functions.
Example 3.2.4 (again):
Let us consider the spaceD= L2whereLis the set of possible locations for one camera, in Figure 3.7 depicted by the grey line. Also, let the orientation of a camera always be such that the camera always faces the middle of the environment directly, as illustrated by the black cross. Thus, the domain is two dimensional.
Objective function Maximize the volumeλ(·) of the intersection of the field of view of both the cam-eras. A global maximum of λis roughly such that both the cameras are situated where the left camera is right now:
argmax
a1,a2∈L
λ(a1,a2)=(x1,x1).
Let us place the cameras by the Iteration 3.2.1 starting with the placement of the left camera in the sub-spaceL. The camera moves over to the right camera, since the intersection is at a maximum there, thereby reaching the point (x2,x2). In a second step the right camera should be readjusted but the intersection of both cameras is already at a subspace maximum.
The algorithm has reached a stationary point but it has not reached the global maximum. In fact, from point (x2,x2), there exists a direction in the domain on which the solution can be improved, namely (1,1). Thus, the stationary point cannot be anywhere near alocalmaximum. Placing the cameras in the subspace of (1,1) means moving them simultaneously.
3.2. OPTIMIZATION PROCEDURE UTILIZING A BLOCK COORDINATE ASCENT 97
x1 x2 L
Figure 3.7: Illustrating a stationary point, not a global maximum: The domain is built by two spaces of camera locationsD= L2(grey line). Each camera always faces the cross in the middle of the environment, thus the domain is two dimensional. Placing the cameras such that the inter-section of the field of view of both the cameras is maximized by Iteration 3.2.1 starting with the left camera reaches a stationary point (x1,x1) which is not a global maximum.
If a regular BCA converges, it reaches a particular type of maximum. This type of maximum is called stationary point in this thesis. On a particular type of function, the additively separable functions, a BCA reaches the stationary point in one iteration step. On this function type, a stationary point corresponds to a global maximum. If a function is differentiable, a stationary point on an inner point of the domain corresponds to a local maximum or a saddle point of the domain. If the objective function is not dif-ferentiable, stationary points can additionally be caused by non-differentiable points. Unfortunately, the objective function in Equation (1.2) is neither separable nor differentiable everywhere. In the following sections, a BCA is introduced that converges nevertheless.