Pruning with Reference Sets

Similarity Search

Pruning with Reference Sets

Nikolaus Augsten

nikolaus.augsten@sbg.ac.at

Dept. of Computer Sciences University of Salzburg

http://dbresearch.uni-salzburg.at Version January 18, 2017

Wintersemester 2016/2017

Outline

1 Reference Sets Definition

Upper and Lower Bound

Combined Approach

Reference Sets Definition

Outline

1 Reference Sets Definition

Upper and Lower Bound

Combined Approach

Reference Sets Definition

Reference Sets [?]

Reference sets take advantage of the triangle inequality for metrics.

An extended version of the triangle inequality is:

| δ (a, b) − δ(b, c ) | ≤ δ (a, c ) ≤ δ(a, b) + δ (b, c ),

where δ() is a metric distance function computed between a, b, and c .

Reference Sets Definition

Illustration: Triangle Inequality

a b

c

δ(a, c )<δ (a, b) + δ(b, c )

a b

c

| δ (a, b) − δ(b, c ) | <δ (a, c )

a b c

δ(a, c )=δ (a, b) + δ(b, c )

a c b

| δ (a, b) − δ(b, c ) | =δ (a, c )

Reference Sets Definition

Notation

S ₁ and S ₂ are sets of trees (unordered forests) K ⊆ S ₁ ∪ S ₂ is called the reference set

T _i ∈ S ₁ and T _j ∈ S ₂ are trees

k _l ∈ K , 1 ≤ l ≤ | K | , is a tree of the reference set

Reference Sets Definition

Reference Vector

v _i is a vector

of size | K |

that stores the distance from T _i ∈ S ₁ to each tree k _l ∈ K

the l -th element of v _i stores the edit distance between T _i and k _l : v _il = δ _t (T _i , k _l )

v _j is the respective vector for T _j ∈ S ₂

Reference Sets Definition

Metric Space

Metric space of the reference set:

the elements of the reference set define the basis of a metric space the vector v _k represents tree T _k as a point in this space

v _kl is the coordinate of the point on the l -th axis

Example: Two trees T _i and T _j in the metric space defined by the reference set K = { k ₁ , k ₂ } .

v _{i 1}

v _{j 1} v _{k 1} v _{i 2}

v _{j 2} v _{k 2}

T _j

T _i

Reference Sets Upper and Lower Bound

Outline

1 Reference Sets Definition

Upper and Lower Bound

Combined Approach

Reference Sets Upper and Lower Bound

Upper and Lower Bound

From the triangle inequality it follows that for all 1 ≤ l ≤ | K |

| v _il − v _jl | ≤ δ _t (T _i , T _j ) ≤ v _il + v _jl Upper bound:

δ _t (T _i , T _j ) ≤ min

l ,1 ≤ l ≤| K | v _il + v _jl = u _t (T _i , T _j ) Lower bound:

δ _t (T _i , T _j ) ≥ max

l ,1 ≤ l ≤| K | | v _il − v _jl | = l _t (T _i , T _j ) Approximate join: match all trees with δ _t (T ₁ , T ₂ ) ≤ τ

Upper bound: If u _t (T _i , T _j ) ≤ τ , then T _i and T _j match.

Lower bound: If l _t (T _i , T _j ) > τ , then T _i and T _j do not match.

Reference Sets Upper and Lower Bound

Example: Similarity Join with Reference Sets

S ₁ = { T ₁ , T ₂ , T ₃ } , S ₂ = { T ₄ , T ₅ , T ₆ } , τ = 2 Reference set K = { T ₁ } , 1 ≤ l ≤ | K | = 1 Distances:

T ₁ T ₂ T ₃ T ₄ T ₅ T ₆

T ₁ 0 4 1 1 4 1

T ₂ 0 4 4 1 4

T ₃ 0 1 4 1

T ₄ 0 4 1

T ₅ 0 4

T ₆ 0

Reference Vectors:

v _1,1 = (0), v _2,1 = (4), v _3,1 = (1), v _4,1 = (1), v _5,1 = (4), v _6,1 = (1)

The clusters C ₁ = { T ₁ , T ₃ , T ₄ , T ₆ } and C ₂ = { T ₂ , T ₅ } are well

separated.

Reference Sets Upper and Lower Bound

Reference Set — Tradeoff

Small reference set: efficient reference vector computation

we have to compute | S | distances for each additional tree in the reference set to construct the vectors

for small reference sets the construction of the vectors is cheaper

Large reference set: effective filters

large reference sets make u _t and l _t more effective

thus, once the reference vectors are computed, less distance computations are needed

Where is the optimum?

Reference Sets Upper and Lower Bound

Well Separated Clusters

We cluster the set S = S ₁ ∪ S ₂ .

The clusters are well separated for threshold τ if

trees within a cluster have small distance (within ^τ ₂ )

trees from different cluster have large distance (more than ^3τ ₂ )

Example: Two well separated clusters C ₁ and C ₂ .

≤ ^τ ₂ > ^3τ ₂ ≤ ^τ ₂

C ₁ C ₂

Reference Sets Upper and Lower Bound

Trees within the Same Cluster — Upper Bound Applies

Upper bound: δ _t (T _i , T _j ) ≤ v _il + v _jl = δ _t (T _i , k _l ) + δ _t (k _l , T _j ) Assumption: T _i ∈ S ₁ and T _j ∈ S ₂ are in the same cluster C .

The clusters are well separated.

The cluster C contains a reference tree k _l ∈ K .

In this case v _il ≤ ^τ ₂ and v _jl ≤ ^τ ₂ ⇒ δ _t (T _i , T _j ) ≤ τ . Result: If T _i , T _j , and k _l are in the same cluster

from v _il and v _jl we conclude that T ₁ and T ₂ match thus we need not to compute δ _t (T _i , T _j )

k T _j

T _i

v _jl

δ _t (T _i , T _j ) v _il

τ 2

Reference Sets Upper and Lower Bound

Trees in Different Clusters — Lower Bound Applies

Lower bound: δ _t (T _i , T _j ) ≥ | v _il − v _jl | = | δ _t (T _i , k _l ) − δ _t (k _l , T _j ) | Assumption: T _i ∈ S ₁ is in cluster C ₁ , T _j ∈ S ₂ is in cluster C ₂ .

The clusters are well separated.

The cluster C ₁ contains a reference tree k _l ∈ K .

In this case v _il ≤ ^τ ₂ and v _jl > ^3τ ₂ ⇒ δ _t (T _i , T _j ) > τ .

Result: If T _i and k _l are in the same cluster, but T _j is in a different cluster

from v _il and v _jl we conclude that T ₁ and T ₂ do not match thus we need not to compute δ _t (T _i , T _j )

≤ ^τ ₂ > ^3τ ₂ ≤ ^τ ₂ T _j T _i

k _l

C ₁ v _il C ₂

v _jl

δ _t (T _i , T _j )

Reference Sets Upper and Lower Bound

Optimum — Well Separated Clusters

Optimum:

clusters are well separated

the reference set contains one tree per cluster

Guha et al. [?] find clusters by sampling and estimate a reference set

size.

Reference Sets Combined Approach

Outline

1 Reference Sets Definition

Upper and Lower Bound

Combined Approach

Reference Sets Combined Approach

Combined Approach

We combine the previous approaches:

lower bound with traversal strings

upper bound with constrained edit distance reference sets

Instead of one vector v _i we compute two vectors for each tree T _i :

lower bound: v _i ^l contains the traversal string distance

upper bound: v _i ^u contains the constrained edit distance

Reference Sets Combined Approach

Metric Space

Metric space of the reference set:

the elements of the reference set define the basis of a metric space each tree T _k is represented by a rectangle in this metric space

v _k ^l and v _k ^u are two opposite corners of this rectangle

Example: Two trees T _i and T _j in the metric space defined by the reference set K = { k ₁ , k ₂ } .

v

v

v

v

v

T

T

Reference Sets Combined Approach

Combined Approach: Triangle Inequality

The triangle equations changes as follows:

(a) For all 1 ≤ l ≤ | K |

δ _t (T _i , T _j ) ≤ v _il ^u + v _jl ^u

(b) For all 1 ≤ l ≤ | K |

δ _t (T _i , T _j ) ≥

 



 

v _jl ^l − v _il ^u if v _jl ^l > v _il ^u v _il ^l − v _jl ^u if v _il ^l > v _jl ^u

0 otherwise

Note:

If v _jl ^l > v _il ^u or v _il ^l > v _jl ^u then [v _il ^l , v _il ^u ] and [v _jl ^l , v _jl ^u ] are disjoint intervals.

In all other cases we can not give a lower bound (other than 0).

Reference Sets Combined Approach

Combined Approach: Upper and Lower Bounds

Upper bound:

u _t (T _i , T _j ) = min

l ,1 ≤ l ≤| K | v _il ^u + v _jl ^u

Lower bound:

l _t (T _i , T _j ) = max

l ,1 ≤ l ≤| K |

 



 

v _jl ^l − v _il ^u if v _jl ^l > v _il ^u v _il ^l − v _jl ^u if v _il ^l > v _jl ^u

0 otherwise

Reference Sets Combined Approach

Illustration: Cases for the Lower Bound

Case v _jl ^l > v _il ^u : lower bound is v _jl ^l − v _il ^u

v _kl v _il ^l v _il ^u v _jl ^l v _jl ^u

lower bound Case v _il ^l > v _jl ^u : lower bound is v _il ^l − v _jl ^u

v _kl v _jl ^l v _jl ^u v _il ^l v _il ^u

lower bound

All other cases ([v _il ^l , v _il ^u ] and [v _jl ^l , v _jl ^u ] overlap): no lower bound

v _kl v _jl ^l

v _il ^l

v _jl ^u

v _il ^u