In this section we characterise of the gradient of a convex function at a point minimising the function over a closed convex set. The following developments form the basis of the local convergence analysis of the optimisation algorithm considered in Chapter 4.
The next lemma can be seen as an extension of the Farkas lemma as it is stated in [Ber09, Proposition 2.3.1].
Lemma B.1 (Farkas lemma) Let A ⊂ Rn such that one of the following conditions holds: Ais nite, or cone(A)is closed. Then{x|a′x ≤0,∀a ∈ A} and cone(A) are closed cones that are polar to each other.
Proof We proceed as in [Ber09] and rst observe that the two considered sets are closed. Indeed, {x|a′x ≤ 0,∀a ∈ A} is a cone, and closed as the intersection of closed halfspaces. IfA is nite, then cone(A)is closed (see the comment following Denition B.1). Hence cone(A)is closed in any case under our assumption. It follows from Proposition B.5-(i) that
(cone(A))∗ = A∗ (B.5)= {x|a′x ≤ 0,∀a∈ A}. (B.6)
The next result can be seen as a local characterisation of a closed convex set from the perspective of any boundary point. It is based on Proposition 3.1, which states that any closed convex set can be characterised by the collection of closed halfspaces that contain the set. In the following lemma, it is sug-gested to regard a closed convex set as the intersection of a (nite or innite) collection of closed halfspaces limited by hyperplanes, so that the set of nor-mal vectors of the hyperplanes supporting the considered set at any boundary point generates a closed conethis condition is met in particular when the set of normal vectors at the boundary point is nite, as for instance on the boundary of any polyhedron, which can be dened as the intersection of a nite collection of halfspaces.
Lemma B.2 (Existence of a characteristic set of hyperplanes)
Let X be a closed, convex subset of Rn. Then one can nd a set A⊂ (Rn\ {0})×R satisfying the following two conditions.
(i) We have X = {x|a′x ≤b,∀(a, b) ∈ A}.
(ii) At each point xin the boundary of X, the setA(x)¯ ⊂Rn\{0} dened by A(x) =¯ {a|(a, a′x)∈ A} is such that eitherA(x)¯ is nite or cone( ¯A(x)) is closed.
Moreover, if x is a boundary point of X, then (X −x)∗ = cone( ¯A(x)). Proof From Proposition 3.1, we know that X = {x|a′x ≤ b,∀(a, b) ∈ H}, where
H = {(a, b)∈ (Rn\ {0})×R|a′x ≤ b,∀x ∈ X} (B.7) is a representation of the collection of all the halfspaces that contain the closed convex set X. Now, let x be a boundary point of X. The set
H¯(x) ={a|(a, a′x) ∈ H} (B.8)
179
is, by construction, the set of the normal vectors of all the supporting hyper-planes to X at x. It follows from the discussion following Denition B.2 that (X −x)∗ = {0} ∪H¯(x), and that {0} ∪H¯(x) is a cone and closed. Hence cone( ¯H(x)) is closed. That H satises (i) and (ii) proves the rst part of the lemma.
Consider now a setA ⊂(Rn\{0})×Rfor which (i) and (ii) hold. We have X−x ={y|a′(x+y)≤ b,∀(a, b)∈ A}. Since (X−x)∗ is the cone generated by the set of the normal vectors of the supporting hyperplanes to (X −x) at the origin, i.e. the hyperplanes {y|a′y ≤ 0} with a ∈ A(x)¯ , (X −x)∗ does not depend on the halfspaces of the type {y|a′(x+ y) ≤ b} with (a, b) ∈ H and a′x < b. Therefore we can write
(X −x)∗ = {y|a′y ≤ 0,∀a ∈ A(x)¯ }∗ (B.9)
= cone( ¯A(x)) (B.10)
where (B.10) follows from Lemma B.1.
We are now able to characterise, at any boundary point of a closed convex set, the gradient of a convex function which achieves a global optimum at this point. The next proposition takes the form of necessary and sucient optimality conditions at boundary points.
Proposition B.6 (Optimality condition for boundary points) Let X be a closed convex subset of Rn, f : Rn → R∪ {∞} a dierentiable con-vex function, and A ⊂ (Rn \ {0})×R a characterisation of X in terms of halfspaces satisfying the requirements (i) and (ii) in Lemma B.2. Consider a point x ∈ dom(f) on the boundary of X and the set A(x)¯ ⊂ Rn\ {0} de-ned as in Lemma B.2 by A(x) =¯ {a|(a, a′x) ∈ A}. The point x is a global minimum of f over X i −∇f(x) is a nonnegative combination of vectors of A(x)¯ .
Proof It is easily seen from Denition B.2 that x satises the rst-order optimality condition (3.22) holds i −∇f(x)∈ (X −x)∗. Using Lemma B.2, we have (X −x)∗ = cone( ¯A(x)), where cone( ¯A(x)) is by denition the set of the nonnegative combinations of vectors of A(x)¯ , which completes the proof.
Extension to interior points. Notice that Lemma B.2 and Proposition B.6 are mostly concerned with boundary points. Besides, for any pointxin the in-terior of a closed convex set X, we have(X−x)∗ = {0} andH¯(x) =∅, where H¯(x)is dened as in the proof of Lemma B.2. This last result is incompatible with the notion of a generated cone, which is only dened for nonempty sets
for the reason that nonnegative combinations of vectors require at least one vector. Since the rst-order condition for a point x in the interior of a closed convex setX to minimise a dierentiable convex functionf overX is given by
∇f(x) = 0, Lemma B.2 and Proposition B.6 can easily be extended to interior points by dening cone(∅) = {0}, and by considering that the nonnegative combination of zero vector is 0.
Case of a polyhedral feasible set. A particular case is when the closed convex set X is a polyhedron of the type (4.44) specied by a nite set C = {c(1), ..., c(r)} of scalar ane constraints Rn → R, i.e. X = {x ∈ Rn|c(x)≤ 0, c ∈ C}. An obvious choice for the set A introduced in Proposition B.6 is to set
A= {(∇c(0),−c(0))|c∈ C} (B.11) where, for c ∈ C, ∇c denotes the gradient of c, which is a constant function equal to the normal vector of c(x) ≤ 0 for all x ∈ Rn. At each boundary point x, we nd A(x) =¯ {∇c(0)|c∈ C, c(x) = 0}. The optimality condition at x reduces to
− ∇f(x)∈ cone( ¯A(x)), (B.12) which holds i −∇f(x) is a nonnegative combination of vectors of A(x)¯ .
Optimality conditions for convex optimisation problems with explicit con-straints. Consider now Problem 3.2 introduced in Section 3.4.2 as the convex optimisation problem in standard form, and letS ≡ Rn. Iff0 is dierentiable, then the conditions of Proposition B.6 are met. Suppose, in addition, that the inequality constraint functions f1, ..., fq are also dierentiable, and let D = ∩pi=0dom(fi). The feasible set of the problem
X = {x|fi(x)≤0, hj(x) = 0, i = 1, ..., q, j = 1, ..., r} (B.13) is a closed subset of D, and convex as per Proposition B.4 in Appendix B.1.
Under certain regularity conditions called constraint qualications and dis-cussed in Section 3.4.3, it is possible to characterise the feasible set X by a setAof halfspaces normal to the gradients of the constraint functions, so that the conditions (i) and (ii) of Lemma B.2 are satised. One of these conditions, the Slater condition, requires the existence of a strictly feasible point, i.e. a point of the feasible set where the non-ane inequality constraints hold with strict inequalities. Other conditions are concerned with the linear indepen-dence of the gradients of the constraint functions. The concept of problem regularity is illustrated in Examples B.1 and B.2.
181
Recall that for i = 1, ..., r and any x ∈ Rn, the equality hi(x) = 0 can be rewritten as hi(x) ≤0 and hi(x)≥ 0. Under a constraint qualication, a
Example B.1 (Constraint qualication (i))
q x2
x1 0
✫✪
✬✩
aq
✫✪
✬✩
aq✛ ✲
Consider the convex optimisation problem minimise
(x1,x2)∈R2
f(x1, x2) subject to (x1−a)2+x22≤a2
(x1+a)2+x22≤a2
(B.14)
wheref :R2→Ris a convex function, andaa positive scalar. One sees easily that the feasible set of this problem is the singletonX ={(0,0)}. The set of halfspaces containing X, given by (B.7), reduces toH = (R2\ {(0,0)})×R≥0, and the set (B.8) of the supporting hyperplanes toX at(0,0) to H¯((0,0)) = R2\ {(0,0)}, with cone( ¯H((0,0))) = R2 closed. Thus, H satises the conditions of Lemma B.2, and it follows from Proposition B.6 that (0,0) is a solution i ∇f(0,0) ∈ R2. Hence (0,0) is the unique solution of the problem for anyf.
The gradients of the two inequality constraint functions, given by 2(x1−a, x2) and 2(x1+a, x2), are respectively equal to2a(−1,0)and2a(1,0)at the solution{(0,0)}, where we note that the (non-ane) inequality constraints are all active and the gradient of the inequality constraint functions are neither linearly independent nor positive-linearly independent, i.e. the Mangasarian-Fromovitz constraint qualication does not hold. The point(0,0)is not a regular point since the construction of a setA satisfying the condition (i) in Lemma B.2 based on the gradients of the constraint functions is in this problem impossible. Indeed, −∇f(0,0) can generally not be expressed as a nonnegative linear combination of2a(−1,0)and2a(1,0). In other words, the KKT conditions are not necessary for this problem.
Example B.2 (Constraint qualication (ii))
q x2
x1 0
❅
❅
❅
❅
❅
❅
❅
✒
❅❅❘
Notice that the optimisation problem (B.14) is equivalent to minimise
(x1,x2)∈R2
f(x1, x2) subject to x1−x2= 0
x1+x2= 0
(B.15)
where the absence of non-ane inequality constraints guaranties that the regularity conditions are met. Moreover, the gradient of the constraints, equal to (1,−1) and (1,1), are positive-linearly independent. Hence the Mangasarian-Fromovitz constraint qualication holds, and(0,0)is regular.
Also, the setA={((1,−1),0),((1,1),0)} satises (i) and (ii) in Lemma B.2. The KKT conditions are now necessary and sucient, and satised at the solution(0,0).
possible choice for the set A is given by A(i)∪A(e), where we dene
A(i) = ∪qi=1{(∇fi(x),∇fi(x)′x)|fi(x) = 0}, (B.16) A(e) = ∪ri=1{(∇hi(0),−hi(0)),(−∇hi(0),−hi(0))}, (B.17) and ∇hi is a constant function (i= 1, ..., r).
At any point x on the boundary of X (seen as a subset of Rn), we nd A(x) = ¯¯ A(i)(x)∪A¯(e)(x)∪(
−A¯(e)(x))
, where
A¯(i)(x) =∪qi=1{∇fi(x)|fi(x) = 0}, (B.18) A¯(e)(x) =∪ri=1{∇hi(x)|hi(x) = 0}. (B.19) The optimality condition at x reduces to ∇f0(x) ∈ cone( ¯A(x)), which holds i −∇f(x) is a nonnegative combination of vectors of A(x)¯ , or equivalently, i one can nd scalars y1, ..., yq and z1, ..., zr satisfying
yi ≥0, i = 1, ..., q, (B.20) yi∇fi(x) = 0, i= 1, ..., q, (B.21)
∇f(x) +∑q
i=1yi∇fi(x) +∑r
i=1zi∇hi(x) = 0. (B.22) Moreover, the condition that x be a feasible point of X reduces to
fi(x)≤0, i= 1, ..., q, (B.23) hi(x) = 0, i= 1, ..., r. (B.24) In the present framework, notice that any interior point of the feasible setX is a point of X where no supporting hyperplane can be found (A(x) =¯ ∅), i.e.
a pointx ∈ X such thatfi(x) < 0fori= 1, ..., q andhi(x) ̸= 0fori= 1, ..., r, which is only possible when the problem has no equality constraints (r = 0).
In that case, the optimality of x is equivalent to ∇f(x) = 0, and it is easily seen that settingr = 0in (B.23)-(B.24) and (B.20)-(B.22) yields the feasibility and optimality conditions for the interior point x, respectively.
All in all, we nd in (B.20)-(B.24) the necessary and sucient conditions for any pointx ∈ D to be a solution of Problem 3.2, and recover the Karush-Kuhn-Tucker conditions for the problem, which were already stated in Propo-sition 3.6. In other words, PropoPropo-sition B.6 can be seen as an expression of the KKT necessary and sucient optimality conditions of the convex optimisation problem for a larger class of convex optimisation problems where the feasi-ble sets are not necessarily specied by a collection of dierentiafeasi-ble convex inequality constraint functions and ane equality constraint functions. We refer for instance to [BSS93] for an extensive discussion on the KKT conditions and the constraint qualications also based on the concept of cones.
183