Learning Assembly Sequence Plans from Vision

ASSEMBLYsome miscellaneous parts or a bolt will follow. Here, however, there is another nut adjacent to the green cube. Thus, the last fallback must be taken into account and another instance of ASSEMBLY-ANCHOR is generated in the corresponding search tree node. This time, only the blue bolt remains as a possible starting point for subcluster analysis because the other objects of the original set of candidates are already integrated into structural descriptions. The blue bolt and an obligatory fallback thus yield two alternatives for further processing. The first alternative yields an assembly consisting of the bolt and the rhomb-nut that may serve as a ^BOLT-PART. Pursuing this assumption yields that the cube adjacent to the rhomb-nut is part of a current assembly which therefore is conjoined with the incoming one. Afterwards, there are no objects left to be examined and the analysis terminates.

1

BAR1 CUBE BOLT₁

BOLT₂

Figure 3.16.: A sequence to assemble the familiar assembly A₁.

Let A be a partially ordered set. An element a∈A is amaximal element if there is no b∈A, b6=asuch that avb. An element a∈A is called aminimal element if for no b∈A, b6=a, bva. Ais called a chainif for alla, b∈A, avborbva. Note that every subset of a chain is a chain. Given a partially ordered setA and a chainC ⊆A, we say thatC is amaximal chain inAif

6 ∃C⁰⊆A:C ⊂C⁰.

I.e. a chain is maximal if it is not a subset of any other chain.

Now that we know about partially ordered sets and chains consider the following Definition:Let o1 and o2 be two elementary objects such that either one of them is a boltBOLT_i and the other is attached to it, or none of them is a bolt but both are connected toBOLT_i. o₁ carrieso₂ on BOLT_i if one of the following conditions holds:

1. o1=o2, or

2. o₁ isBOLT_i and o₂ is attached to it, or

3. o₁ was attached to BOLT_i ereo₂ was attached to it.

Ifo₁ carries o₂ on BOLT_i we will henceforth writeo₁vci o₂.¹⁰

From this definition we see that each bolt in a bolted assembly generates a relation that imposes a partial order on the set of compound objects. Moreover, such orders will always be chains.¹¹However, the name of the relation seems a little odd. We will motivate it by means of the familiar assemblyA1. SupposeA1 is assembled as shown in Fig. 3.16.

First, BAR₁ is attached to BOLT₁. If oriented as sketched in the figure, BOLT₁ will carry BAR₁ and we may writeBOLT₁ vc1 BAR₁. After attaching CU BE₁ to the bolt and in a corresponding orientation, BOLT1 and BAR1 will both carry CU BE1. Thus, will also have BOLT₁ vc¹ CU BE₁ and BAR₁ vc¹ CU BE₁. The dependency on the orientation is of course undesirable. Therefore, the relation x carries y on z should be understood abstractly. It is meant to express that, with respect to z, xprecedes y or x

10Appendix C.2 motivates why the relation is chosen with respect to a bolt.

11See Appendix C.2 for a proof.

BOLT₁

1 BOLT₂

BAR

CUBE₁

(a)

BOLT¹vc1 BAR¹vc1CU BE¹ BOLT2vc2CU BE1

(b)

Figure 3.17.: A graphical representation of the ordering relations in A1 and the two maximal chains contained in the assembly.

(a)

ASSEMBLY(Highest Level) BOLT2

ASSEMBLY(Complex Nut) BOLT1

BAR1 (BOLT1,a) () ()

CUBE1 (BOLT1) () (BOLT2) ()

(b)

Figure 3.18.: Calculated positions of mating features and a high-level assembly sequence plan for the assembly A₁.

is below y. Finally, if BOLT₂ is attached to the assembly, it may be seen as a carrier of the cube as well and we will have BOLT₂ v_c2 CU BE₁. But as it neither is in direct contact with BAR₁ nor with BOLT₁, it should not be interpreted to carry those.

The ordering relations within assembly A1 can be graphically summarized as shown in Fig. 3.17(a). We easily recognize two maximal chains; Fig. 3.17(b) shows them using the notation we just introduced.

Since bolted assemblies necessarily consist of at least two objects, each chain in an assembly has a minimal element b and a maximal element n. Objectb will always be a bolt andnwill be a nut. Thus, asb6=n, these elements can be thought to define a line.

Hence, lines defined by sequentially arranged objects are a general topological prop-erty of bolted assemblies. The basic idea behind our heuristic was that similar structures must be found in images ofbaufix^rassemblies since photography preserves topology. And indeed, there are approximately linear arrangements: the center points of image regions associated to bolted objects define lines in the image plane along which syntactic analysis

RING1 RNUT1

BOLT3

BOLT1

BAR1

CUBE2 BOLT2

CUBE1

(a)

ASSEMBLY(Highest Level) ASSEMBLY(Complex Bolt)

BOLT1

RNUT1 (BOLT1) RING1 (BOLT1)

ASSEMBLY(Complex Misc) BOLT3

BAR1 (BOLT3,a) () () () (BOLT1,a) CUBE2 (BOLT3) () () ()

ASSEMBLY(Complex Nut) BOLT2

CUBE1 (BOLT2) () (BOLT1) ()

(b)

Figure 3.19.: The assembly known from Fig. 3.1(b) and a possible sequence plan. The syntactic component structure of this plan corresponds to the detection result discussed in Appendix B.

can be performed. For objects like rhomb-nuts or rings these center points are assumed to represent the projected position of the corresponding mating feature. For bars this assumption is certainly invalid. However, their mating features are assumed to be equally spaced along the principle axis of the corresponding image region. And in fact, this yields positions which comply with the postulated linear arrangements. Dealing with cubes the center point will likewise not represent a mating feature. Thus, mating features of cubes are assumed to be found at the points of intersection between the estimated lines and the border of a cube’s region (see Fig. 3.18(a)).

Given their image coordinates, it is possible to estimate which mating features par-ticipate in a connection. Object attributes defined in the Ernestnetwork for assembly detection can verify whether a feature belongs to one of the lines that are due to a partial order. If so, the feature is assumed to be attached to the corresponding bolt. The detec-tion algorithm can therefore generate descripdetec-tions of assemblies which describe hierar-chical component structures and simultaneously reflect relations among mating features.

Data structures that include both kinds of information correspond to what Chakrabarty and Wolter [24] call high-level assembly sequence plans. Examples of sequence plans for baufix^rassemblies generated from visual inspection are shown in Figs. 3.18 and 3.19.

The plan in Fig. 3.19 results from the detection process discussed in Appendix B. Its overall structure reflects, that certain complex parts of the assembly can be constructed in parallel (cf. Fig. 3.1(a)). Objects with holes occur with (lists of) slots representing their mating features. Although cubes have six surfaces, in the plan they only have four slots since baufix^rcubes feature only four threaded holes. If a feature is occupied, it contains the name of the corresponding bolt. Slots of bars are additionally marked

with a flag that indicates whether the corresponding bolt is inserted from above (a) or below (b). These flags are defined intrinsically: each instance of a bar has an attribute to register the temporal order in which the bar’s mating features were considered during cluster analysis. The bolt in the first of these holes is assumed to be above the bar.

Other bolts associated with that bar are above as well if they are on the same side of its principle axis. Those on the other side are said to be below. The order of the slots of bars and cubes is also defined intrinsically. Holes of a cube, for example, are listed in a clockwise manner: the hole that was considered first during analysis will always appear as the leftmost slot, the one intrinsically left to it is represented by the second slot, its vis-a-vis corresponds to the third slot, and the hole on its right appears last in the list.

Further details on this labeling scheme can be found in [13].

Plans like this characterize how the elementary assembly objects are connected to each other. Syntactic component structures augmented with information of mating fea-ture relations thus yield topologically unique descriptions of an assembly¹². High-level plans derived from vision therefore may be translated into appropriate natural language or manipulator instructions in subsequent steps of processing.