Semantic Networks as Parsers

In earlier work [12] we presented the idea to implement the recursive model of baufix^rassemblies as a semantic network using Ernest[71, 113].

Ernest knows three different types of nodes and three different types of links. Con-cepts represent classes of objects, events, or abstract conceptions with certain general properties. They form the basis for semantical pattern analysis. Dealing with image un-derstanding, for instance, interest lies in interpreting the sensor signal by means of given concepts. An instance is a copy of a concept where descriptions of common properties of a class are substituted by values derived from the signal. In an intermediate state of processing, instances may not be computable because certain prerequisites are missing.

Nevertheless, the available information can be used to constrain an uninstantiated con-cept. This is done via the node type modified concept. Like in other semantic network formalismspartlinks decomposes a concept into its natural components. If a concept has several possible component structures, they can be modeled as differentmodalities. An-other well-known kind of links arespecializations. They realize an inheritance mechanism

by which a concept inherits all properties of the general one. To distinguish knowledge from different levels of abstraction Ernest introduces concrete links. Moreover, con-cepts are associated with attributes which represent numerical or symbolic features of the modeled term.Relationsdefining constraints for attributes can be specified and must be satisfied before valid instances can be generated. Since the results from initial seg-mentation seldom are perfect, concepts can also contain ajudgmentfunction to estimate the congruence between an image area and the term represented by the concept. The knowledge modeled in anErnestnetwork is utilized in processes of creating and assess-ing modified concepts and instances. Assumassess-ing that complex object recognition means to detect all object parts, an object is considered to be recognized as soon as instances for all its parts were generated.

With all these features Ernest is a powerful tool for pattern recognition. Within the scope of this thesis it is interesting to note that Ernest networks with no other than part links are structurally equivalent to context free grammars⁸. Due to this equiv-alence the idea to implement generic assembly models as semantic networks is perfectly reasonable. It is in fact advantageous since semantic networks provide a way to deal with constraints among syntactic primitives that is missing in simple CFGs. Attributes assigned to concepts may represent knowledge or restrictions of non context free nature and thus provide opportunities known from attributed grammars (cf. [30]).

Processing Model

Our assembly model was mapped into anErnestknowledge base and attributes were de-fined which represent context sensitive and domain specific knowledge like, for example, the state of an object’s mating features. This actually solves a problem mentioned earlier:

the CFG that modelsbaufix^rassemblies allows to derive physically infeasible structures.

TheErnestimplementation prevents this undesirable effect because attributes that reg-ister mating relations among objects suppress the generation of structures corresponding to infeasible attachments. Further attributes were introduced to support the control al-gorithm that governs the instantiation process. They register, for instance, which objects were already examined and estimate which objects should be considered next.

Given the semantic network as the processing model and the results from elementary object recognition as input data, detectingbaufix^rassemblies is a process of sequentially analyzing object clusters. Resulting hierarchical assembly structures are visualized as shown in Fig. 3.10(a). The figure shows a white polygon nested in black one, both surrounding images regions that were recognized to depict assemblies. The white one corresponds to a subassembly of the complex object enclosed by the black polygon;

Figure. 3.10(b) depicts the corresponding syntactic structure.

8This is easy to see if we consider the AND/OR graph representation of a CFG: nodes correspond to concepts, AND links correspond to part links, and each hyperlink corresponds to a modality. A formal proof can be found in Appendix C.1.

(a)

BOLT

BOLT-PART NUT-PART

CUBE BAR

MISC-PART ASSEMBLY BOLT-PART

BOLT

ASSEMBLY

NUT-PART

(b)

Figure 3.10.: Assembly detection result cast into the image and hierarchical structure corresponding to this result.

Details on the process of instantiating concepts from the knowledge base can be found in Appendix B.2. Here, we will only roughly sketch the underlying ideas: Since bolts are obligatory components of every baufix^rassembly, analyzing a 2D cluster starts with a region representing a bolt. After instantiating the bolt as the bolt-part of an assembly, an attribute marks its image region as ‘considered’ and the adjacent objects are examined.

If one of them complies with a criterion for choosing objects for the next step of the analysis and depicts a miscellaneous object the corresponding concept is instantiated and its region is marked ‘considered’ as well. This process iterates until a cube or a rhomb-nut are found such that an assembly can be instantiated. If an assembly was found but the cluster still contains unmarked objects, it is determined what role the assembly may play within a more complex unit, then it is instantiated as the corresponding functional part of another assembly, and finally the analysis is continued respectively.

In order to obtain a consistent syntactic interpretation for a 2D cluster its ele-ments must be examined in a certain sequence. Generally, a method to order higher-dimensionally distributed primitives for parsing is domain dependent and has to be found heuristically. In the baufix^rdomain, however, simple topological considerations avoid misinterpretations. Objects that are put onto bolts are linearly arranged in space, thus, as Fig. 3.11(a) underlines, the centers of mass of their image regions are ordered ap-proximately linear as well. This defines directions in the image plane which, if estimated at the beginning of an analysis or after a (sub)assembly has been instantiated, can be used to search for further objects⁹. Hence, objects to be instantiated in a (sub)assembly

9The center of a bolt region and the center of an adjacent region, for example, define an initial search direction. If there are several neighbors, the control algorithm must consider alternative directions in a backtracking manner if one of them fails to yield a good interpretation.

(a)

2 1

(b)

Figure 3.11.: 3.11(a) Positions of the centers of mass of the bolts and the cube and calculated positions of holes of the bar. 3.11(b) The directions for sequential cluster analysis yielding the result in Fig. 3.10.

must be adjacent to each other and if there are several neighbors, one of them should comply with the current search direction. Objects deviating from these directions are not assumed to belong to the current (sub)assembly. If an adjacent object depicts a bar the positions of its holes have to be considered instead of its center point and since holes used in a connection are not visible, their positions are calculated by dividing the bar’s main axis into equidistant ranges.

Arrow 1 in Fig. 3.11(b) shows the initial search direction for our example. It is given by the center of the blue bolt’s head and the position of the nearest hole of the bar.

Arrow 2 in Fig. 3.11(b) indicates the direction for searching after a subassembly has been found that serves as a nut. It is defined by the center of a cube that was assumed to be attached to another bolt and a corresponding reference point of an unmarked neighboring object. Further details on this search heuristic can be found in [9] and [13].