## Nonlinear Optimization I

Dr. Thomas M. Surowiec

Humboldt University of Berlin Department of Mathematics

Summer 2013

## Convexity

Definition

**1** A setX⊂R^{n}is called convex, if for allx,y∈X and allλ∈(0,1)

λx+ (1−λ)y∈X, i.e. the segment[x,y]lies completely inX.

**2** LetX⊂R^{n}be convex. A functionf :X →Ris called

**1** (strictly) convex (inX), if for allx,y ∈X and for allλ∈(0,1)
the following holds true:

f(λx+ (1−λ)y)≤λf(x) + (1−λ)f(y) (f(λx+ (1−λ)y)< λf(x) + (1−λ)f(y))

**3** f is uniformly convex, if there existsµ >0with

f(λx+ (1−λ)y) +µλ(1−λ)||x−y||^{2}≤λf(x) + (1−λ)f(y),

∀x,y∈X,∀λ∈(0,1).

## A Few Remarks

Geometrically speaking convexity of a function implies that the line segment between function values lies above the function’s graph.

Clearly: Uniform convexity→strict convexity→convexity.

## Example

Example

Letf:R^{n}→Rbe a quadratic function, i.e.,
f(x) + 1

2x^{T}Ax+b^{T}x+c
withA∈ Sn,b∈R^{n},c∈R. Then

**1** f is convex iffAis positive semi-definite.

**2** f is strictly convex ifffunif. conv. iffApos. def.

These concepts can be extended in some cases to differentiabe convex functions...

## Supporting Hyperplanes/Quadratic Functions

Lemma

LetX⊂R^{n}be open and convex, andf :X →Rcontinuously differentiable.

Then the following assertions hold

**1** f is convex (onX) iff for allx,y∈Xit holds that
f(x)≥f(y) +∇f(y)^{T}(x−y).

**2** f is strictly convex (onX) iff for allx,y∈X, withx6=yit holds that
f(x)>f(y) +∇f(y)^{T}(x−y).

**3** f is uniformly convex (onX) iff there existsµ >0such that
f(x)≥f(y) +∇f(y)^{T}(x−y) +µ||x−y||^{2}
for allx,y∈X.

Proof.

On the board.

## Second-Order Properties

Theorem

LetX⊂R^{n}be open and convex, andf :X →Rtwice continuously
differentiable. Then the following assertions hold

**1** f is convex (onX) iff∇^{2}f(x)is positive semi-definite for allx∈X.

**2** If∇^{2}f(x)is positive definite for allx∈X, thenfis strictly convex (onX).

**3** f is uniformly convex (onX) iff∇^{2}f(x)is uniformly positive definite on
X, i.e., if there existsµ >0such that

d^{T}∇^{2}f(x)d≥µ||d||^{2}
forx∈X and for alld ∈R^{n}.

Proof.

On the board.

## Existence of Solutions

Lemma

Letf:R^{n}→Rbe continuously differentiable andx^{0}∈R^{n}arbitrary.

Furthermore, assume that the level set
L(x^{0}) :=n

x∈R^{n}

f(x)≤f(x^{0})o

is convex and thatfis uniformly convex inL(x^{0}). Then the setL(x^{0})is
compact.

Proof.

On the board.

## Existence of Solutions

Theorem

Letf:R^{n}→Rbe continuously differentiable andX⊂R^{n}be convex.

Consider the optimization problem min

x∈Xf(x), (1)

then the following statements hold

**1** Iff is convex onX, then the solution set is convex (possibly empty).

**2** Iff is strictly convex onX, then(1)has at most one solution.

**3** Iff is uniformly convex (onX) andXis non-empty and closed, then(1)
has exactly one solution.

Proof.

On the board.

## A Few Cautionary Remarks

**1** f(x) =exp(x)is strictly convex onX =R, yet it has no minimimum!

**2** X must be closed for a solution to exist. Any ideas?

## Consequences I

Lemma

Letf:R^{n}→Rbe continuously differentiable,x^{0}∈R, and let the level set
L(x^{0})be convex andf be uniformly convex onL(x^{0}). In addition, suppose
thatx^{∗}∈R^{n}is the unique global minimizer off. Then there existsµ >0with

µ||x−x^{∗}||^{2}≤f(x)−f(x^{∗}),∀x∈L(x^{0}).

## Consequences II

Theorem

Letf:R^{n}→Rbe a continuously differentiable and convex function. and let
x^{∗}∈R^{n}be a stationary point off. Thenx^{∗}is a global minimizer off inR^{n}.