SOLUTION TECHNIQUES
2.2 Sequential Quadratic Programming and the Trust Region Method of YuanTrust Region Method of Yuan
In this section, we review a trust region algorithm yielding a stationary point for the continuous nonlinear program (2.1). To simplify readability we assume that the box constraints are included in the nonlinear constraintsg(x), see (2.5). It was proposed by Yuan [112] and is closely related to the well-known sequential quadratic programming methods. The most common nonlinear programming algorithms are interior point methods, e.g., W¨achter and Biegler [105] or sequential quadratic programming (SQP) methods, e.g., Schittkowski and Yuan [96]. These nonlinear programming algorithms converge globally towards a stationary point. A global minimum can only be guaran-teed for convex problems, e.g., the continuous relaxation of the convex MINLP (1.6).
Nonlinear programming problems naturally arise during the solution process of convex mixed-integer nonlinear optimization problems (1.6), e.g., as continuous relaxation or by fixing the integer variablesy∈Nni. We focus on the trust region method proposed by Yuan [112], since it is the base for a novel MINLP solution method presented in Section 3.1.
Every SQP method is based on solving a series of continuous quadratic (QP) sub-problems. In iteration k a quadratic subproblem is constructed by linearizing the constraints at the current iterate xk. The objective function is a quadratic approx-imation of the Lagrangian function L(x, λ, µ) of NLP (2.1), see Definition 2.4 and Definition 2.7 for a suitable choice of the matrixBkforming the quadratic term in the objective function. The subproblem in iterationk is given by
d∈Rn:
min ∇xf(xk)Td+12dTBkd
s.t. gj(xk) +∇xgj(xk)Td = 0, j=1, . . . , me, gj(xk) +∇xgj(xk)Td ≥ 0, j=me+1, . . . , m.
(2.27)
The solution of QP (2.27) is denoted by dk. The next iterate xk+1 is obtained either directly, i.e.,xk+1 =xk+dk, if a trust region constraintd≤ k∆kkis included, where∆k denotes the trust region radius. Or it is obtained by a line search, i.e.,xk+1 =xk+αkdk, with αk ∈(0, 1] reducing the step-lengthαk until a sufficient descent with respect to an appropriate merit-function is obtained.
The subproblems of the trust region method of Yuan [112] approximate theL∞-penalty function introduced in Definition 2.9 instead.
Definition 2.9. TheL∞-penalty functionPσ(x)associated with the penalty parameter σ∈R+ and x ∈Rn is given by
Pσ(x) = f(x) +σkg(x)−k∞, (2.28)
with
g(x)− := (g1(x)−, . . . , gm(x)−)T (2.29) gj(x)− :=
gj(x), j=1, . . . , me
min{gj(x), 0}, j=me+1, . . . , m. (2.30) In each iterationkthe trust region method constructs a model of the original problem based on the current iterate xk. Typically, the accuracy and therefore the quality of the model decreases with an increasing distance fromxk. To control the quality of the model, the maximal distance from the current iterate is restricted by the trust region radius ∆k.
In iteration k Yuan’s trust region algorithm approximates the L∞-penalty function leading to the following problem:
d∈Rn :
min Φk(d)
s.t. kdk∞ ≤ ∆k,
(2.31)
where the objective function is given by
Φk(d) := ∇x f(xk)Td+1
2dTBkd
+ σkk(g(xk) + [∇x g(xk)]Td)−k∞.
(2.32)
Note, that problem (2.31) is equivalent to a quadratic program, see Yuan [112]. There-fore, it can be solved efficiently by any QP solver, e.g., QL of Schittkowski [94]. The optimal solution of subproblem (2.31) is denoted by dk. It provides a search direction to determine the next iterate, i.e.,
xk+1 = xk+dk. (2.33)
Problem (2.31) predicts a decrease in theL∞-penalty function. We define
Predk:= Φk(0) −Φk(dk). (2.34) If the subsequent stopping criterion
dk=0 (2.35)
and certain assumptions hold, the current iterate xk is one out of three stationary points, that are defined below. One of these stationary points corresponds to a KKT-point of NLP (2.1), see Yuan [112].
To reduce the constraint violation a penalty parameter is introduced, that penalizes the violation of constraints. If feasibility is not sufficiently improved, the penalty
parameterσkhas to be increased. The increase of the penalty parameter is controlled by a parameterδk ∈R with δk> 0.
Comparing the predicted reduction given by (2.34), with the realized decrease of the L∞-penalty function, the quality of model associated with problem (2.31) can be judged. The corresponding measure is denoted by
rk := Pσk(xk) −Pσk(xk+dk)
Predk . (2.36)
If the model corresponding to problem (2.31) is of poor quality, e.g.rk≤0, the trust region radius has to be decreased. Otherwise, the trust region radius can be extended.
Subsuming all components we can specify the trust region method of Yuan [112].
Algorithm 2.1. (Trust Region Method of Yuan)
1. Given x0 ∈ Rn, ∆0 > 0, B0 ∈ Rn×n symmetric and positive definite, δ0 > 0, σ0 > 0 and k:=0.
Evaluate the functions f(x0) and g(x0) and determine gradients ∇xf(x0) and
∇xg(x0).
2. Solve subproblem (2.31) determining dk.
If stopping criterion (2.35) is satisfied, then STOP.
Else evaluate f(xk+dk), g(xk+dk) and Pσk(xk+dk).
Evaluate the quality of the model with respect to theL∞-penalty function byrk (2.36).
3. Check for a descent with respect to the L∞-penalty function.
If rk > 0, then update iterate, i.e., set xk+1 :=xk+dk. Else set xk+1 :=xk, Bk+1:= Bk, ∆k+1:= 14kdkk∞, k:= k+1.
GOTO Step 2.
4. Adapt trust region radius:
∆k+1 :=
max{∆k, 4kdkk∞}, if rk > 0.9,
∆k, if 0.9≥rk≥0.1, min{14∆k,12kdkk∞}, if rk < 0.1.
(2.37)
Choose Bk+1, such that Bk+1 is any symmetric, positive definite matrix.
Penalty Update:
If
Φk(0) −Φk(dk) ≤ σkδkmin
∆k,kg(xk)−k∞ , (2.38) then set
σk+1 := 2σk, (2.39)
δk+1 := 1
4δk. (2.40)
Else set
σk+1 := σk, (2.41)
δk+1 := δk. (2.42)
5. Compute ∇x f(xk+1) and [∇x g(xk+1)], set k:=k+1 and GOTO Step 2.
The trust region method of Yuan converges towards a stationary point, if Assump-tion 2.1 is satisfied, see Yuan [112] and Jarre and Stoer [66].
Assumption 2.1. 1. f(x) and gj(x), j=1, . . . , m are continuously differentiable
∀x∈Rn.
2. The sequences {xk} and {Bk} are bounded ∀ k.
Moreover, we review the theoretical properties of Algorithm 2.1, according to Yuan [112].
The limit of kg(xk)−k∞ exists, if the sequence of penalty parameter {σk} goes to in-finity.
Lemma 2.1. If Assumption 2.1 holds and σk→ ∞, then lim
k→∞kg(xk)−k∞ exists.
Proof. See Yuan [112].
Algorithm 2.1 converges towards one of three different stationary points, see Yuan [112].
These are either a KKT-point for problem (2.1) as specified in Theorem 2.1. Or it is a infeasible stationary point or a singular stationary point.
Yuan [112] defines an infeasible stationary point as follows.
Definition 2.10. Let F be the feasible region of NLP (2.1). x∗ ∈ Rn\F is called an infeasible stationary point, if
1. kg(x∗)−k∞ > 0 2. min
d∈Rnk(g(x∗) + [∇x g(x∗)]Td)−k∞=kg(x∗)−k∞
holds.
An infeasible stationary point is a minimizer of the infinity norm of the linearized constraints, see Yuan [112]. In contrast to an infeasible stationary point, a singular stationary point is feasible. It is defined as follows.
Definition 2.11. LetFbe the feasible region of NLP (2.1).x∗ ∈Fis called a singular stationary point, if the following conditions hold.
1. kg(x∗)−k∞=0.
2. There exists a sequence{^xk}converging towards x∗ such thatkg(^xk)−k∞> 0and
klim→∞ min
kdk∞≤kg(^xk)−k∞
k(g(^xk) + [∇x g(^xk)]Td)−k∞
kg(^xk)−k∞ =1. (2.43) In addition a stationary point is defined as follows.
Definition 2.12. Let F be the feasible region of NLP (2.1). x∗ ∈ F is called a sta-tionary point, if the following conditions hold.
1. kg(x∗)−k∞=0.
2. If
∇xgj(x∗)Ts ≥ 0, j ∈J> (2.44)
∇xgj(x∗)Ts = 0, j ∈J= (2.45) hold for s∈Rn, then
∇xf(x∗)Ts ≥ 0 (2.46)
is satisfied
Note, that a stationary point according to Definition 2.12 corresponds to a KKT-point of NLP (2.1), see Jarre and Stoer [66].
The following theorem shows, that the trust region method of Yuan converges towards a KKT point, if the sequence of penalty parameters {σk}is bounded.
Theorem 2.3. If Assumption 2.1 holds and the sequence of penalty parameters {σk} is bounded, one member of the sequence of iterates {xk} is a stationary point speci-fied in Definition 2.12 or the sequence {xk} generated by Algorithm 2.1 possesses an accumulation point that is a stationary point.
Proof. See Yuan [112].
If the sequence of penalty parameters {σk} is unbounded and the constraints are violated in the limit, Yuan’s trust region method converges towards an infeasible stationary point given by Definition 2.10.
Lemma 2.2. If Assumption 2.1 holds, lim
k→∞σk =∞and lim
k→∞kg(xk)−k∞> 0, then the sequence{xk}possesses an infeasible stationary point of NLP (2.1)as an accumulation point.
Proof. See Yuan [112].
If the sequence of penalty parameters {σk}is unbounded but the constraint violation goes to zero, Yuan’s trust region method converges towards a singular stationary point as specified in Definition 2.11.
Lemma 2.3. If Assumption 2.1 holds, lim
k→∞σk = ∞ and lim
k→∞kg(xk)−k∞ = 0, then the sequence {xk} generated by Algorithm 2.1 possesses a singular stationary point of NLP (2.1) as an accumulation point.
Proof. See Yuan [112].