Numerical Optimization

Numerical Optimization

CHAPTER 13. ADMM

Today

ADMM: Alternating Direction Method of Multipliers

[Glowinski, R. & Marroco, A., 1975]

[D. Gabay & B. Mercier, 1976]

For an introduction: [Boyd et al., FnT ML, 2010]

Aim:

•   Understanding the algorithm

•   Convergence

•   Consensus formulation

Dual Ascent

min

x ∈ R ⁿ

f ( x )

s . t

. Ax = b

A ∈ R ^{m × n} , f : R ⁿ → R convex

Lagrangian L ( x ; y ) = f ( x ) + y ^T ( Ax − b ) y ∈ R ^m

Dual objective function: g ( y ) = inf

x

L ( x ; y )

Let x ^∗ minimizes L (x ; y ) for a given y . Then

g (y ) = L (x ^∗ ; y ) = f (x ^∗ ) + y ^T (Ax ^∗ − b)

Dual Problem

max

y

g ( y ) ^{Ax ∗ − b ∈ ∂ g ( y ) ,}

x ^∗ ∈ arg min

x

L ( x ; y )

Using subgradient ascent:

α ^k > 0 is a stepsize

x ^{k +1} ∈ arg min

x

L (x ; y ^k )

y ^{k +1} = y ^k + α ^k ( Ax ^{k +1} − b )

Dual Subgradient Ascent

When alpha ^k is chosen carefully, and with additional assumptions, this procedure can produce

However, this requires conditions often do not hold in practice.

x ^{k +1} ∈ arg min

x

L ( x ; y ^k )

y ^{k +1} = y ^k + α ^k ( Ax ^{k +1} − b )

x ^k → x ^∗ , y ^k → y ^∗

Augmented Lagrangian

Lagrangian:

Augmented Lagrangian:

This is the Lagrangian function associated with an equivalent problem:

L ( x ; y ) = f ( x ) + y ^T ( Ax − b )

L _ρ ( x ; y ) = f ( x ) + y ^T ( Ax − b ) + ρ

2 k Ax − b k ² ₂ ρ > 0 : penalty parameter

min

x

f ( x ) + ρ

2 k Ax − b k ²

s . t . Ax = b

MM: Method of Multipliers

Dual ascent with augmented Lagrangian:

•   Converges in more general conditions than dual ascent

x ^{k +1} ∈ arg min

x

L _ρ ( x ; y ^k )

y ^{k +1} = y ^k + ρ(Ax ^{k +1} − b)

ADMM

min

x ∈ R ⁿ , z ∈ R ^m f ( x ) + g ( z ) s . t . Ax + Bz = c

A ∈ R ^{p × n} , B ∈ R ^{p × m}

Augmented Lagrangian:

L _ρ ( x , z ; y ) = f ( x ) + g ( z ) + y ^T ( Ax + Bz − c ) + ρ

2 k Ax + Bz − c k ² ₂

“Alternating direction” method of multipliers

x ^{k +1} ∈ arg min

x

L _ρ ( x , z ^k ; y ^k )

z ^{k +1} ∈ arg min

z

L _ρ ( x ^{k +1} , z ; y ^k )

y ^{k +1} = y ^k + ρ( Ax ^{k +1} + Bz ^{k +1} − b )

Coordinate-Wise Minimization

Dual Ascent

MM vs. ADMM

x ^{k +1} ∈ arg min

x

L _ρ ( x , z ^k ; y ^k )

z ^{k +1} ∈ arg min

z

L _ρ ( x ^{k +1} , z ; y ^k )

y ^{k +1} = y ^k + ρ( Ax ^{k +1} + Bz ^{k +1} − b )

( x ^{k +1} , z ^{k +1} ) ∈ arg min

x

L ρ ( x , z ; y ^k )

y ^{k +1} = y ^k + ρ ( Ax ^{k +1} + Bz ^{k +1} − b )

MM

ADMM

Convergence of ADMM

Assumptions:

1. f : R ⁿ → R ∪ {∞}, g : R ^m → R ∪ {∞} are closed, proper, and convex

This implies that x-update and z-update are solvable, i.e., minimizers exist (but may not be unique)

epi f = {( x , t ) ∈ R ⁿ × R : f ( x ) ≤ t }

⇔

epi g = {( x , t ) ∈ R ⁿ × R : g ( x ) ≤ t }

are closed nonempty convex sets

Assumption 2:

With assumption 1, this implies that

L 0 has a saddle point, i.e. there exists (x ^∗ , z ^∗ , y ^∗ ) s.t.

L 0 (x ^∗ , z ^∗ , y ) ≤ L 0 (x ^∗ , z ^∗ , y ^∗ ) ≤ L 0 (x , z , y ^∗ ), ∀ x , z , y

(x ^∗ , z ^∗ ) is a primal solution of min

x , z f (x ) + g (z ) s.t. Ax + Bz = c

y ^∗ is a dual solution

Convergence of ADMM

Under assumptions 1 & 2,

r ^k := Ax ^k + Bz ^k − c → 0 as k → ∞ f ( x ^k ) + g ( z ^k ) → f ^∗ + g ^∗ as k → ∞

y ^k → y ^∗ as k → ∞

Residual Objective Dual variable

Primal variables need not converge to optimal values, although such

results can be shown under additional assumptions

Convergence of ADMM

General convex case

•   Sublinear convergence O(1/k)

•   [He & Yuan, SIAM J Numerical Analysis, 2012]

Strongly convex case

•   Linear convergence

•   [Deng & Yin, Rice Univ. Tech rep, TR12-14, 2012]

Global Variable Consensus

x min ∈ R ⁿ f ( x ) =

N

!

i =1

f _i ( x ) f _i : R ⁿ → R ∪ {∞} convex

A global variable x is shared across f _i ’s

x _i ∈ R min ⁿ , z ∈ R ⁿ

N

!

i =1

f _i ( x _i )

s.t. x _i − z = 0 , i = 1 , 2 , . . . , N .

A simple reformulation (global consensus problem):

ADMM for Global Consensus

Augmented Lagrangian:

L _ρ (x 1 , . . . , x _N , z ; y ) =

N

!

i =1

"

f _i (x _i ) + y _i ^T (x _i − z ) + ρ

2 "x _i − z " ² ₂ # ADMM:

x _i ^{k +1} = arg min

x

! f _i ( x _i ) + ( y _i ^k ) ^T ( x _i − z ^k ) + ρ

2 " x _i − z ^k " ² ₂ "

z ^{k +1} = 1 N

N

#

i =1

$

x _i ^{k +1} + 1 ρ

y _i ^k

%

k +1 k k +1 k +1

Simplification

⇒ z ^{k +1} = ¯ x ^{k +1} + 1 ρ y ¯ ^k

¯

x := 1 N

N

!

i =1

x _i

⇒ by averaging, ¯ y ^{k +1} = ¯ y ^k + ρ(¯ x ^{k +1} − z ^{k +1} )

∴ y ¯ ^{k +1} = 0

∴ z ^{k +1} = ¯ x ^{k +1}

ADMM:

x _i ^{k +1} = arg min

x

!

f _i (x _i ) + (y _i ^k ) ^T (x _i − z ^k ) + ρ

2 "x _i − z ^k " ² ₂ "

z ^{k +1} = 1 N

N

#

i =1

$

x _i ^{k +1} + 1 ρ

y _i ^k

%

y _i ^{k +1} = y _i ^k + ρ ( x _i ^{k +1} − z ^{k +1} )

Simplification

z ^{k +1} = ¯ x ^{k +1}

ADMM:

x _i ^{k +1} = arg min

x

!

f _i ( x _i ) + ( y _i ^k ) ^T ( x _i − z ^k ) + ρ

2 " x _i − z ^k " ² ₂ "

z ^{k +1} = 1 N

N

#

i =1

$

x _i ^{k +1} + 1 ρ y _i ^k

%

y _i ^{k +1} = y _i ^k + ρ ( x _i ^{k +1} − z ^{k +1} )

Simplified ADMM:

x _i ^{k +1} = arg min

x i

⇣

f _i ( x _i ) + ( y _i ^k ) ^T ( x _i − x ¯ ^k ) + ρ

2 k x _i − x ¯ ^k k ² ₂ ⌘

ADMM for Global Consensus

Each function access local data

Simplified ADMM:

x _i ^{k +1} = arg min

x _i

⇣

f _i ( x _i ) + ( y _i ^k ) ^T ( x _i − x ¯ ^k ) + ρ

2 k x _i − x ¯ ^k k ² ₂ ⌘ y _i ^{k +1} = y _i ^k + ρ( x _i ^{k +1} − x ¯ ^{k +1} )

x ₁ x ₂

x _N

. . .

. . .

y

f 1

f 2

. . .

f _N

x 1

x 2

x _N

¯ x x ¯

¯ x

Message-Passing Algorithm

•   Communication cost?

•   Data storage?