Planning and Optimization D8. M&S: Strategies and Label Reduction Gabriele R¨oger and Thomas Keller

(1)

D8. M&S: Strategies and Label Reduction

Gabriele R¨oger and Thomas Keller

Universit¨at Basel

November 7, 2018

(2)

Content of this Course

Planning

Classical

Tasks Progression/

Regression Complexity Heuristics

Probabilistic

MDPs Uninformed Search

Heuristic Search Monte-Carlo

Methods

(3)

Content of this Course: Heuristics

Heuristics

Delete Relaxation

Abstraction

Abstractions in General

Pattern Databases

Merge &

Shrink Landmarks

Potential Heuristics Cost Partitioning

(4)

Merging Strategies

(5)

Content of this Course: Merge & Shrink

Merge & Shrink

Synchronized Product Merge & Shrink Algorithm

Heuristic Properties Strategies Label Reduction

(6)

Generic Algorithm Template

Generic M&S computation algorithm abs := {T^π^{v} |v ∈V}

while abs contains more than one abstract transition system:

select A₁,A₂ fromabs

shrink A₁ and/or A₂ untilsize(A₁)·size(A₂)≤N abs := abs\ {A₁,A₂} ∪ {A₁⊗ A₂}

returnthe remaining abstract transition system in abs Remaining question:

Which abstractions to select? merging strategy

(7)

Linear Merging Strategies

Linear Merging Strategy

In each iteration after the first, choose the abstraction computed in the previous iteration asA₁.

Rationale: only maintains one “complex” abstraction at a time Fully defined by an ordering of atomic projections.

(8)

Linear Merging Strategies: Choosing the Ordering

Use similar causal graph criteria as for growing patterns.

Example: Strategy ofh_HHH

h_HHH: Ordering of atomic projections Start with a goal variable.

Add variables that appear in preconditions of operators affecting previous variables.

If that is not possible, add a goal variable.

Rationale: increases h quickly

(9)

Non-linear Merging Strategies

Non-linear merging strategies only recently gained more interest in the planning community.

One reason: Better label reduction techniques (later in this chapter) enabled a more efficient computation.

Examples:

DFP: preferrably merge transition systems that must synchronize on labels that occur close to a goal state.

UMCandMIASM: Build clusters of variables with strong interactions and first merge variables within each cluster.

Each merge-and-shrink heuristic computed with a non-linear merging strategy can also be computed with a linear merging strategy.

However, linear merging can require a super-polynomial blow-up of the final representation size.

(10)

Shrinking Strategies

(11)

Content of this Course: Merge & Shrink

Merge & Shrink

(12)

Generic Algorithm Template

Generic M&S computation algorithm abs := {T^π^{v} |v ∈V}

while abs contains more than one abstraction:

select A₁,A₂ fromabs

shrink A₁ and/or A₂ untilsize(A₁)·size(A₂)≤N abs := abs\ {A₁,A₂} ∪ {A₁⊗ A₂}

returnthe remaining abstraction in abs

N: parameter bounding number of abstract states Remaining Questions:

Which abstractions to select? merging strategy How to shrink an abstraction? shrinking strategy

(13)

Shrinking Strategies

How to shrink an abstraction?

We cover two common approaches:

f-preserving shrinking bisimulation-based shrinking

(14)

f -preserving Shrinking Strategy

f-preserving Shrinking Strategy

Repeatedly combine abstract states with identicalabstract goal distances (h values) and identicalabstract initial state distances (g values).

Rationale: preserves heuristic value and overall graph shape Tie-breaking Criterion

Prefer combining states whereg+h is high.

In case of ties, combine states whereh is high.

Rationale: states with high g+h values are less likely to be explored by A^∗, so inaccuracies there matter less

(15)

Bisimulation

Definition (Bisimulation)

LetT =hS,L,c,T,s₀,S_?ibe a transition system. An equivalence relation∼onS is a bisimulation for T if for every hs, `,s⁰i ∈T and everyt ∼s there is a transition ht, `,t⁰i ∈T with t⁰ ∼s⁰. A bisimulation∼isgoal-respectingif s ∼t implies that either s,t ∈S? or s,t6∈S?.

(16)

Bisimulation: Example

1

2

3

4

5

o p

o

o p

q o q

o

p

∼with equivalence classes {{1,2,5},{3,4}} is a goal-respecting bisimulation.

(17)

Bisimulations as Abstractions

Theorem (Bisimulations as Abstractions)

LetT =hS,L,c,T,s₀,S_?ibe a transition system and ∼be a bisimulation forT. Then α∼:S → {[s]∼|s ∈S} with α∼(s) = [s]∼ is an abstraction ofT .

Note: [s]∼ denotes the equivalence class of s.

Note: Surjectivity follows from the definition of the codomain Note: as the image ofα∼.

(18)

Abstractions as Bisimulations

Definition (Abstraction as Bisimulation)

LetT =hS,L,c,T,s0,S?ibe a transition system and α:S →S⁰ be an abstraction ofT. The abstraction induces the equivalence relation∼_α as s ∼_αt iffα(s) =α(t).

We say thatα is a (goal-respecting) bisimulation forT if ∼_α is a (goal-respecting) bisimulation forT.

(19)

Abstraction as Bisimulations: Example

Abstractionα with

α(1) =α(2) =α(5) =A andα(3) =α(4) =B is a goal-respecting bisimulation forT.

T

1

2

3

4

5

o p

o

o p

q o q

o

p

T^α

A B

o p

o,q

(20)

Goal-respecting Bisimulations are Exact (1)

Theorem

Let X be a collection of transition systems. Letα be an

abstraction forT_i ∈X . If α is a goal-respecting bisimulation then the transformation from X to X⁰ := (X \ {T_i})∪ {T_i^α}is exact.

Proof.

LetT_X =T₁⊗ · · · ⊗ T_n=hS,L,c,T,s₀,S_?iand w.l.o.g.

T_X⁰ =T₁⊗ · · · ⊗ T_i−1⊗ T_i^α⊗ T_i+1⊗ · · · ⊗ T_n=hS⁰,L⁰,c⁰,T⁰,s₀⁰,S_?⁰i.

Considerσ(hs₁, . . . ,sni) =hs₁, . . . ,si−1, α(si),si+1, . . . ,sni for the mapping of states andλ= id for the mapping of labels.

1 Mappings σ andλsatisfy the requirements of safe transformations becauseα is an abstraction and we have chosen the mapping functions as before.

. . .

(21)

Goal-respecting Bisimulations are Exact (2)

Proof (continued).

2 Ifhs⁰, `,t⁰i ∈T⁰ with s⁰ =hs₁⁰, . . . ,s_n⁰iand t⁰ =ht₁⁰, . . . ,t_n⁰i, then for j 6=i transition system T_j has transition hs_j⁰, `,t_j⁰i (*) andT_i^α has transition hs_i⁰, `,t_i⁰i. This implies that T_i has a transition hs_i⁰⁰, `,t_i⁰⁰i for somes_i⁰⁰∈α⁻¹(s_i⁰) andt_i⁰⁰ ∈α⁻¹(t_i⁰).

As α is a bisimulation, there must be such a transition for all such s_i⁰⁰ andt_i⁰⁰ (**).

Each s ∈σ⁻¹(s⁰) has the forms =hs₁, . . . ,sniwith sj =s_j⁰ for j 6=i ands_i ∈α⁻¹(s_i⁰). Analogously for each

t =ht₁, . . . ,tni ∈σ⁻¹(t⁰). From (*) and (**) follows that T_j has a transition hs_j, `,t_ji for all j ∈ {1, . . . ,n}, so for each such s andt,T contains the transition hs, `,ti.

. . .

(22)

Goal-respecting Bisimulations are Exact (3)

Proof (continued).

3 For s_?⁰ =hs₁⁰, . . . ,s_n⁰i ∈S_?⁰, eachs_j⁰ with j 6=i must be a goal state of T_j (*) and s_i⁰ must be a goal state of T_i^α. The latter implies that at least on s_i⁰⁰∈α⁻¹(s_i⁰) is a goal state ofT_i. As α is goal-respecting, all states from α⁻¹(s_i⁰) are goal states of T_i (**).

Considers?=hs₁, . . . ,sni ∈σ⁻¹(s_?⁰). By the definition of σ, s_j =s_j⁰ for j 6=i ands_i ∈α⁻¹(s_i⁰). From (*) and (**), each s_j (j ∈ {1, . . . ,n}) is a goal state of T_j and, hence, s_? a goal state of T_X.

4 As λ= id and the transformation does not change the label cost function, c(`) =c⁰(λ(`)) for all`∈L.

(23)

Bisimulations: Discussion

As all bisimulations preserve all relevant information, we are interested in the coarsest such abstraction (to shrink as much as possible).

There is always a unique coarsest bisimulation for T and it can be computed efficiently (from the explicit representation).

In some cases, computing the bisimulation is still too

expensive or it cannot sufficiently shrink a transition system.

(24)

Greedy Bisimulations

Definition (Greedy Bisimulation)

LetT =hS,L,c,T,s₀,S_?ibe a transition system. An equivalence relation∼onS is a greedy bisimulationfor T if it is a bisimulation for the systemhS,L,c,T^G,s₀,S_?i, where

T^G ={hs, `,ti | hs, `,ti ∈T,h^∗(s) =h^∗(t) +c(`)}.

Greedy bisimulation only considers transitions that are used in an optimal solution of some state ofT.

(25)

Greedy Bisimulation is h-preserving

Theorem

LetT be a transition system and letα be an abstraction ofT. If

∼_α is a goal-respecting greedy bisimulation forT then h_T^∗α =h_T^∗.

(Proof omitted.)

Note: This does not mean that replacing T with T^α in a collection of transition systems is a safe transformation! Abstractionα preserves solution costs “locally” but not “globally”.

(26)

Label Reduction

(27)

Content of this Course: Merge & Shrink

Merge & Shrink

(28)

Label Reduction: Motivation (1)

T

5

o,o⁰ p

o

o p

q

o,o⁰ q

o

p⁰

T⁰

o,o⁰

o,o⁰,p,p⁰,q

Whenever there is a transition with labelo⁰ there is also a

transition with labelo. If o⁰ is not cheaper than o, we can always use the transition witho.

Idea: Replace o ando⁰ with label o⁰⁰ with cost of o

(29)

Label Reduction: Motivation (2)

T

s t

o⁰⁰ p

o⁰⁰

o p

q

o⁰⁰ q

o⁰⁰

p⁰

T⁰

o⁰⁰

o⁰⁰,p,p⁰,q

Statess and t are not bisimilar due to labelsp andp⁰. In T⁰ they label the same (parallel) transitions. Ifp and p⁰ have the same cost, in such a situation there is no need for distinguishing them.

Idea: Replace p andp⁰ with labelp⁰⁰ with same cost.

(30)

Label Reduction: Motivation (3)

T

s t

o⁰⁰ p⁰⁰

o⁰⁰

p⁰⁰ o

q

o⁰⁰ q

o⁰⁰

p⁰⁰

T⁰

o⁰⁰ o⁰⁰,p⁰⁰,q

Label reductions reduce the time and memory requirement for merge and shrink steps and enable coarser bisimulation abstractions.

When is label reduction a safe transformation?

(31)

Label Reduction: Definition

Definition (Label Reduction)

LetX be a collection of transition systems with label setLand label cost functionc. Alabel reductionhλ,c⁰i for X is given by a functionλ:L→L⁰, whereL⁰ is an arbitrary set of labels, and a label cost functionc⁰ onL⁰ such that for all`∈L,c⁰(λ(`))≤c(`).

ForT =hS,L,c,T,s0,S?i ∈X thelabel-reduced transition system isT^hλ,c⁰ⁱ=hS,L⁰,c⁰,{hs, λ(`),ti | hs, `,ti ∈T},s0,S?i.

Thelabel-reduced collectionis X^hλ,c⁰ⁱ ={T^hλ,c⁰ⁱ| T ∈X}.

L⁰∩L6=∅ andL⁰ =Lare allowed.

(32)

Label Reduction is Safe (1)

Theorem (Label Reduction is Safe)

Let X be a collection of transition systems andhλ,c⁰i be a

label-reduction for X . Thetransformation from X to X^hλ,c⁰ⁱ is safe.

Proof.

We show that the transformation is safe, usingσ= id for the mapping of states andλfor the mapping of labels.

The label cost function ofT_Xhλ,c0i is c⁰ and has the required property by the definition of label reduction. . . .

(33)

Label Reduction is Safe (2)

Theorem (Label Reduction is Safe)

Let X be a collection of transition systems andhλ,c⁰i be a

label-reduction for X . Thetransformation from X to X^hλ,c⁰ⁱ is safe.

Proof (continued).

By the definition of synchronized products,T_X has a transition hhs₁, . . . ,s|X|i, `,ht₁, . . . ,t|X|ii if for alli,T_i ∈X has a transition hs_i, `,tii. By the definition of label-reduced transition systems, this implies thatT^hλ,c⁰ⁱ has a corresponding transitionhs_i, λ(`),t_ii, so

T_Xhλ,c0i has a transitionhs, λ(`),ti=hσ(s), λ(`), σ(t)i(definition

of synchronized products).

For each goal states_? ofT_X, state σ(s_?) =s_? is a goal state of

T_Xhλ,c0i because the transformation replaces each transition system

with a system that has the same goal states.

(34)

More Terminology

LetX be a collection of transition systems with labelsL. Let

`, `⁰ ∈Lbe labels and letT ∈X.

Label `isalivein X if allT⁰ ∈X have some transition labelled with `. Otherwise, `is dead.

Label `locally subsumeslabel `⁰ in T if for all transitions hs, `⁰,ti ofT there is also a transitionhs, `,tiin T.

` globally subsumes`⁰ if it locally subsumes`⁰ in all T⁰ ∈X.

` and`⁰ are locally equivalentin T if they label the same transitions in T, i.e. `locally subsumes`⁰ in T and vice versa.

` and`⁰ are T-combinable if they are locally equivalent in all transition systems T⁰ ∈X \ {T }.

(35)

Exact Label Reduction

Theorem (Criteria for Exact Label Reduction)

Let X be a collection of transition systems with cost function c and label set L that contains no dead labels.

Lethλ,c⁰i be a label-reduction for X such thatλcombines labels

`1 and`2 and leaves other labels unchanged. The transformation from X to X^hλ,c⁰ⁱ is exact iff c(`1) =c(`2), c⁰(λ(`)) =c(`) for all

`∈L, and

`₁ globally subsumes`₂, or

`2 globally subsumes`1, or

`1 and`2 are T-combinable for some T ∈X . (Proof omitted.)

(36)

Back to Example (1)

T

5

o,o⁰ p

o

o p

q

o,o⁰ q

o

p⁰

T⁰

o,o⁰

o,o⁰,p,p⁰,q

Label o globally subsumes label o⁰.

(37)

Back to Example (2)

T

s t

o⁰⁰ p

o⁰⁰

o p

q

o⁰⁰ q

o⁰⁰

p⁰

T⁰

o⁰⁰

o⁰⁰,p,p⁰,q

Labels p and p⁰ are T-combinable.

(38)

Computation of Exact Label Reduction (1)

For given labels`1, `2, the criteria can be tested in low-order polynomial time.

Finding globally subsumed labels involves finding subset relationsships in a set family.

no linear-time algorithms known

The following algorithm exploits only T-combinability.

(39)

Computation of Exact Label Reduction (2)

eq_i := set of label equivalence classes ofT_i ∈X Label-reduction based onT_i-combinability

eq:={L}

for j ∈ {1, . . . ,|X|} \ {i}

Refine eq with eq_j

// two labels are in the same set of eq

// iff they are locally equivalent in all T_j 6=T_i. λ= id

for B ∈eq

samecost := {[`]∼_c |`∈B, `⁰ ∼_c `⁰⁰ iffc(`⁰) =c(`⁰⁰)}

for L⁰ ∈samecost

`_new := new label

c⁰(`new) := cost of labels in L⁰ for `∈L⁰

λ(`) =`_new

(40)

Application in Merge-and-Shrink Algorithm

Generic M&S Computation Algorithm with Label Reduction abs := {T^π^{v} |v ∈V}

while abs contains more than one abstract transition system:

select T₁,T₂ from abs

possibly label-reduce all T ∈abs

(e.g. based onT₁- and/or T₂-combinability).

shrink T₁ and/or T₂ untilsize(T₁)·size(T₂)≤N possibly label-reduce all T ∈abs

abs := abs\ {T₁,T₂} ∪ {T₁⊗ T₂}

returnthe remaining abstract transition system in abs

(41)

Summary

(42)

Summary

Bisimulationis an exactshrinking method.

There is a wide range of merging strategies. We only covered some important ones.

Label reductionis crucial for the performance of the

merge-and-shrink algorithm, especially when using bisimilarity for shrinking.

(43)

Literature

(44)

Literature (1)

References on merge-and-shrink abstractions:

Klaus Dr¨ager, Bernd Finkbeiner and Andreas Podelski.

Directed Model Checking with Distance-Preserving Abstractions.

Proc. SPIN 2006, pp. 19–34, 2006.

Introducesmerge-and-shrink abstractions (for model-checking) andDFPmerging strategy.

Malte Helmert, Patrik Haslum and J¨org Hoffmann.

Flexible Abstraction Heuristics for Optimal Sequential Planning.

Proc. ICAPS 2007, pp. 176–183, 2007.

Introduces merge-and-shrink abstractionsfor planning.

(45)

Literature (2)

Raz Nissim, J¨org Hoffmann and Malte Helmert.

Computing Perfect Heuristics in Polynomial Time: On Bisimulation and Merge-and-Shrink Abstractions in Optimal Planning.

Proc. IJCAI 2011, pp. 1983–1990, 2011.

Introducesbisimulation-based shrinking.

Malte Helmert, Patrik Haslum, J¨org Hoffmann and Raz Nissim.

Merge-and-Shrink Abstraction: A Method for Generating Lower Bounds in Factored State Spaces.

Journal of the ACM 61 (3), pp. 16:1–63, 2014.

Detailedjournal versionof the previous two publications.

(46)

Literature (3)

Silvan Sievers, Martin Wehrle and Malte Helmert.

Generalized Label Reduction for Merge-and-Shrink Heuristics.

Proc. AAAI 2014, pp. 2358–2366, 2014.

Introduceslabel reductionas covered in these slides (there has been a more complicated version before).

Gaojian Fan, Martin M¨uller and Robert Holte.

Non-linear merging strategies for merge-and-shrink based on variable interactions.

Proc. AAAI 2014, pp. 2358–2366, 2014.

IntroducesUMC and MIASM merging strategies