Summer2013 Dr.ThomasM.Surowiec NonlinearOptimizationI

Nonlinear Optimization I

Dr. Thomas M. Surowiec

Humboldt University of Berlin Department of Mathematics

Summer 2013

Convexity

Definition

1 A setX⊂Rⁿis called convex, if for allx,y∈X and allλ∈(0,1)

λx+ (1−λ)y∈X, i.e. the segment[x,y]lies completely inX.

2 LetX⊂Rⁿbe convex. A functionf :X →Ris called

1 (strictly) convex (inX), if for allx,y ∈X and for allλ∈(0,1) the following holds true:

f(λx+ (1−λ)y)≤λf(x) + (1−λ)f(y) (f(λx+ (1−λ)y)< λf(x) + (1−λ)f(y))

3 f is uniformly convex, if there existsµ >0with

f(λx+ (1−λ)y) +µλ(1−λ)||x−y||²≤λf(x) + (1−λ)f(y),

∀x,y∈X,∀λ∈(0,1).

A Few Remarks

Geometrically speaking convexity of a function implies that the line segment between function values lies above the function’s graph.

Clearly: Uniform convexity→strict convexity→convexity.

Example

Example

Letf:Rⁿ→Rbe a quadratic function, i.e., f(x) + 1

2x^TAx+b^Tx+c withA∈ Sn,b∈Rⁿ,c∈R. Then

1 f is convex iffAis positive semi-definite.

2 f is strictly convex ifffunif. conv. iffApos. def.

These concepts can be extended in some cases to differentiabe convex functions...

Supporting Hyperplanes/Quadratic Functions

Lemma

LetX⊂Rⁿbe open and convex, andf :X →Rcontinuously differentiable.

Then the following assertions hold

1 f is convex (onX) iff for allx,y∈Xit holds that f(x)≥f(y) +∇f(y)^T(x−y).

2 f is strictly convex (onX) iff for allx,y∈X, withx6=yit holds that f(x)>f(y) +∇f(y)^T(x−y).

3 f is uniformly convex (onX) iff there existsµ >0such that f(x)≥f(y) +∇f(y)^T(x−y) +µ||x−y||² for allx,y∈X.

Proof.

On the board.

Second-Order Properties

Theorem

LetX⊂Rⁿbe open and convex, andf :X →Rtwice continuously differentiable. Then the following assertions hold

1 f is convex (onX) iff∇²f(x)is positive semi-definite for allx∈X.

2 If∇²f(x)is positive definite for allx∈X, thenfis strictly convex (onX).

3 f is uniformly convex (onX) iff∇²f(x)is uniformly positive definite on X, i.e., if there existsµ >0such that

d^T∇²f(x)d≥µ||d||² forx∈X and for alld ∈Rⁿ.

Proof.

On the board.

Existence of Solutions

Lemma

Letf:Rⁿ→Rbe continuously differentiable andx⁰∈Rⁿarbitrary.

Furthermore, assume that the level set L(x⁰) :=n

x∈Rⁿ

f(x)≤f(x⁰)o

is convex and thatfis uniformly convex inL(x⁰). Then the setL(x⁰)is compact.

Proof.

On the board.

Existence of Solutions

Theorem

Letf:Rⁿ→Rbe continuously differentiable andX⊂Rⁿbe convex.

Consider the optimization problem min

x∈Xf(x), (1)

then the following statements hold

1 Iff is convex onX, then the solution set is convex (possibly empty).

2 Iff is strictly convex onX, then(1)has at most one solution.

3 Iff is uniformly convex (onX) andXis non-empty and closed, then(1) has exactly one solution.

Proof.

On the board.

A Few Cautionary Remarks

1 f(x) =exp(x)is strictly convex onX =R, yet it has no minimimum!

2 X must be closed for a solution to exist. Any ideas?

Consequences I

Lemma

Letf:Rⁿ→Rbe continuously differentiable,x⁰∈R, and let the level set L(x⁰)be convex andf be uniformly convex onL(x⁰). In addition, suppose thatx^∗∈Rⁿis the unique global minimizer off. Then there existsµ >0with

µ||x−x^∗||²≤f(x)−f(x^∗),∀x∈L(x⁰).

Consequences II

Theorem

Letf:Rⁿ→Rbe a continuously differentiable and convex function. and let x^∗∈Rⁿbe a stationary point off. Thenx^∗is a global minimizer off inRⁿ.