The Interpretation Process

DLI is a process that iterates between abductive and deductive reasoning in order to explain a set of observations extracted from media by SLI.

DLI can be adapted to the application needs in two different ways. First, it can be adapted to determine which observations require an explanation, called fiats and repre-sented with Γ₂, and which observations are believed to be true by default, called bona fide and represented with Γ₁. Thus, the set of observations (Γ) provided by SLI processes is divided into two sets Γ₁ and Γ₂. Second, DLI can be adapted to determine which cri-teria for hypothesis selection should be applied. These two aspects are represented with two configuration functions, called trustiness (Ψ) and preference (Ω) respectively. The configuration of trustiness (Ψ) represents a “contract” that is specified through the set of rules R. R defines the space of possible explanations. In other words, the space of abducibles. As described in Section 3.2, on ontology and rule design patterns, the design of the ontology and rules should be grounded on media content, and, according to the notion of interpretation inspired by [NM06, NM08, MN08], fiat observations are those assertions that represent configurations of objects.

To recall, in [NM06, NM08, MN08], aggregates are used to express the properties and constraints which make a particular set of objects worth being recognized as a whole. In this way aggregates are representational units that help to explain object configurations (relations between the objects) w.r.t. a domain of knowledge. For example, for the object configuration shown in Figure 2.20 (see page 51) the keyword table top can be used as an explanation, such that the configuration “makes sense” w.r.t. to the domain of table settings. Thus, that a fork is on one side of a plate and a knife on the other side of the same plate is explained given that they are part of atable top.

Provided this notion of aggregates, the expected configuration of trustiness is to con-sider role assertions as fiat assertions. Concept assertions, on the other hand are conside-red as bona fide assertions in this specific application.

Nevertheless, there can be specific reasons to consider concept assertions as fiats as well. This is influenced by the type of media content that is to be interpreted. For example, a textual content with an official-informative character such as news reports, clinical reports, etc., one assumes that if the name of a person is explicit in the text (observed), then there is sufficient evidence to explain the existence of a person and there is no need to explain any relations with other observations, such that a rule that applies in a backward-chaining way can be defined that explains the person name with two assertions, e.g., the instance of a person and a role “has name” that relates the instance of a person, with the instance of the person name observed. In this way, trustiness (Ψ)

is a function that considers every observation γ ∈ Γ₂ as fiat provided a rule head whose predicate matches the predicate of γ ∈Γ₂.

Algorithm 2 The Deep-Level Interpretation (DLI) Algorithm

1: functioninterpret(O,R, Ax,Γ0,Ω,Ψ)

The preference function (Ω), as introduced before in Section 2.5.3, configures the ab-duction algorithm in two respects. First, it is used in the variable substitution process, where priorities over substitution candidates are established. Second, it is used to deter-mine which constraints should be fulfilled to select preferred explanations. In this case the recommended constraints introduced in Section 2.5.2 (see page 2.5.2) can be manipulated.

Having introduced both configuration functions (Ψ and Ω), the interpretation algorithm (see Algorithm 2) is now explained together with its application in the running example.

The main function, namedinterpret, is called for the first time whenever there are SLI results for a specific media objectd_x, such that there is an Abox, here called Γ₀, containing the results of SLI. The function interpret gets as input the following elements: a domain ontology O (see Appendix A), a set of rules R (see Appendix B), an empty Abox A_x which is associated to the media object d_x being interpreted, an Abox Γ₀ containing the observations fromdxobtained by SLI, a preference function Ω, and the trustiness function Ψ.

For the step-by-step example consider the assertions in Figure 2.17 (page 40) and Figure 2.19 (page 41) as content for Γ₀. The first step is to divide Γ₀ into bona fide and fiat observations (lines 3 and 4). For this, trustiness Ψ is applied over the observations Γ0, such that Γ2 is composed of all assertions in Γ0 whose predicate matches a predicate

60 CHAPTER 2. DEEP-LEVEL INTERPRETATION OF TEXT from a rule’s head in R. Γ₁ contains all the assertions in Γ₀ minus Γ₂. Afterwards, Γ₁ is added to the Abox A_x (see line 5). The following example shows the division of Γ₀.

Γ1={date1:StartDate,(date1, “13 August2002”):hasValue, country1:CountryName,(country1, “Russia”):hasValue, hjName1:HighJumpName,(hjName1, “high jump”):hasValue, perf2:Performance,(perf2, “2.26”):hasValue,

country₂:CountryName,(country₂, “Finland”):hasValue, city₁:CityName,(city₁, “Helsinki”):hasValue,

pName1:PersonName,(pName1, “Jaroslav Rybakov”):hasValue, perf1:Performance,(perf1, “2.29”):hasValue,

pName2:PersonName,(pName2, “Oskari Fronensis”):hasValue, rank1:Ranking, (rank1, “won”):hasValue,

rank₂:Ranking, (rank₂, “silver”):hasValue, (pName₁,perf₁):personNameToPerformance, (pName2,perf2):personNameToPerformance, (hjName1,perf1):sportsNameToPerformance, (hjName1,perf2):sportsNameToPerformance}

A_x=A_x∪Γ₁

γ1 = (hjName1,city1):sportsNameToCityName

γ2 = (pName1,country1):personNameToCountryName γ3 = (pName2,country2):personNameToCountryName γ4 = (hjName1,date1):sportsNameToDate

Γ₂={γ1, γ₂, γ₃, γ₄}

The aim of the DLI process is to explain each γ ∈ Γ₂. Notice that before applying the function compute explanations for a given γ, the assertion γ is transferred from the set Γ₂ to A_x. Hence, the interpretation process terminates when there are no more fiats to explain (Γ₂ =∅).

The set of explanations ∆s, for a givenγ, obtained after applying the abduction func-tion (see compute explanations in line 10 and algorithm 1 in page 54) can be empty given the absence of applicable rules in R that produce a consistent explanation. More-over, if explanations are found, then every ∆∈∆s also complies with the recommended constraints given that Ω is used.

Once explanations for a givenγ are obtained (∆s6=∅), then deductive and abductive reasoning is applied to each ∆ ∈ ∆s. Deductive reasoning is executed in the function

Figure 2.22: Graphical representation of a run of the interpretation algorithm.

apply, in order to add new assertions to the set of fiat assertions (Γ₂) provided the new assertions from ∆. Abductive reasoning is applied in the function compute explanations (see Algorithm 1, page 54) for every recursive call of the function interpret. A graphic representation of this algorithm is presented in Figure 2.22.

As described in Section 3.2, the design of the set R distinguishes⁸ between so called abduction rules (see page 89) anddeduction rules (see page 92), such that only deduction rules are applied in a forward-chained way with the function apply.

If more than one explanation is found, a branch for each ∆∈∆s is created, resulting in a set of Aboxes. Each Abox is composed ofA_x =A_x∪∆. Furthermore, a different Γ₂ is

8In RacerPro it is possible to use options to distinguish between rules that apply in a forward-chaining and in a backward-chaining way. The option “:forward-rule-p nil” specifies that a rule applies in a backward-chaining way, and the command “:backward-rule-p nil” specifies that a rule applies in a forward-chaining way.

62 CHAPTER 2. DEEP-LEVEL INTERPRETATION OF TEXT expected provided the different ∆_safter the function apply is called on the corresponding A_x∪∆. Afterwards, a new recursive call to the function interpret is done. This process continues until Γ₂ is empty in each branch. To continue with the interpretation example, considerγ₁

γ1 := (hjN ame1, city1) :sportsN ameT oCityN ame

for which the call to functioncompute explanations results in the following

∆s:={∆1={new1:HighJ umpCompetition,

(new1, hjN ame1) :hasN ame, hjN ame1:HighJ umpN ame, (new₁, city₁) :takesP lace, city₁:CityN ame}}

(already introduced in Section 2.5.3 on abduction). This explanation is added to the Abox A_x. Applying the rules with the function apply to the newly extended Abox, results into two new assertions such that Γ2 looks as follows:

γ2= (pN ame1, country1) :personN ameT oCountryN ame γ3= (pN ame2, country2) :personN ameT oCountryN ame γ4= (hjN ame1, date1) :sportsN ameT oDate

γ5= (new1, perf1) :sportsCompetitionT oP erf ormance γ₆= (new₁, perf₂) :sportsCompetitionT oP erf ormance

Γ2={γ2, γ3, γ4, γ5, γ6}

A new call to the function interpret is done to explain a second γ ∈ Γ2, here called γ2. According to the setR (see Appendix B) only one rule is applicable suth that the call to the function compute explanationsproduces the following explanation:

∆s={∆1={new2:P erson,

(new₂, pN ame₁) :hasN ame, pN ame₁:P ersonN ame,

(new2, country1) :hasN ationality, country1:CountryN ame}}

After adding this explanation to the Abox, applying the rules, with functionapply, creates a new fiat assertion;γ₇ = (new₂, perf₁) :personT oP erf ormance. A new call tointerpret is done and now γ₃ is explained:

∆s={∆1={new3:P erson,

(new₃, pN ame₂) :hasN ame, pN ame₂:P ersonN ame,

(new3, country2) :hasN ationality, country2:CountryN ame}}

Notice that due to number restrictions on the role hasN ame in the description of the concept P erson (see Appendix A, page 145), an instance of person can have at most one role hasN ame. Thus, without these restrictions, the abduction algorithm with the default configuration (Ω) would have used the assertions of the Abox such as new₂ : P erson to explain γ₃ without having to hypothesize a new individual. Af-ter adding this explanation to the Abox, applying the rules creates a new fiat assertion;

γ₈ = (new₃, perf₂) : personT oP erf ormance. In the next iteration γ₄ (see page 62) is explained

∆s={∆1={new1:HighJ umpCompetition,

(new1, hjN ame1) :hasN ame, hjN ame1:HighJ umpN ame, (new₁, date₁) :hasDate, date₁:Date}}

Note that existing assertions from the Abox are used, thus according to Ω the set indi-viduals that already exist are prioritized over the set of new indiindi-viduals (new_i) during substitution (see line 4 of Algorithm 1, page 54). The same process continues for γ₅, γ₆, γ₇ and γ₈ until Γ₂ is empty. The result is an Abox containing both SLI (Γ₁ and Γ₂) as well as DLI results as follows

A_x= Γ₁∪Γ₂ ∪

A graphical display of the Abox extracted by interpretation is presented in Figure 2.23 (see page 64). In this figure it can be observed that individualnew3is not only an instance of P erson, but also an instance ofHighJ umper. This is implicit by the range restriction over the role hasP articipant in the definition of a high jump trial in the Tbox.

The result of applying DLI to a text document is an Abox (or a set of Aboxes depending on the number of interpretations found, see the branches in Figure 2.22 on page 61), here-after called interpretation Abox. This means that for every document object that is ana-lyzed (dx), a corresponding interpretation Abox Ax (or set of Aboxes Axi, i={1, . . . , n}

64 CHAPTER 2. DEEP-LEVEL INTERPRETATION OF TEXT

Figure 2.23: Graphical representation of DLI’s results for the text in Figure 2.8 (page 28 ).

) is found, which contains results from SLI and DLI processes. In order to associate the interpretation Abox A_x with the corresponding document object d_x, a content manage-ment system is required which provides for unique identifiers. In this way, provided a unique identifier for a document object, a structure is required that associates the unique identifier with the corresponding interpretation Abox. Ideally, such a structure has also the form of Abox assertions (see Figure 2.24), with the advantage that a system that queries over interpretation Aboxes, can use the same query language to obtain the unique identifier of the related document object for its later retrieval. Moreover, in order to associate specific SLI results with media segments, reference annotations are used. As described in Section 2.4.2 (see page 33), reference annotations are stored in a relational structure associating them to the corresponding segments of data by spans that identify start and end offsets. Furthermore, classifying media segments w.r.t. content structural aspects is advantageous for further processing for example, to support multimedia fusion (as is described in section 2.6). The research of content structural aspects, e.g., MPEG-7, is beyond this work, but the work proposed by Dasiopoulou et al. in [DDS⁺07] is used in the DLI framework. More specifically, the MCO (Multimedia Content Ontology)⁹ onto-logy is used to represent the content structural aspects of a media object and relate them to SLI and DLI results such that reference annotations are created.

9Downloadable from: http://www.boemie.org/ontologies

The MCO ontology models knowledge about multimedia content structure, which encom-passes the representation of:

• Different types of media, e.g., text, image, web pages, etc.

• Logical relations among types of media, e.g., a web page contains text and two captioned images.

• Decomposition of media into segments, e.g., text is decomposed into phrases, which in turn are decomposed into words and characters.

• Relations between media segments and domain semantics, e.g., segments of a text with SLI and DLI results.

In this way, as Figure 2.24 graphically shows, reference annotations have the form of Abox assertions such that a so calledcontent structure Abox is created for a media object and associated to the corresponding interpretation Abox. In contrast to SLI which is necessarily a media-dependent process, DLI is a media-independent process that can be used for the interpretation of text and image as long as the input meets the requirements represented in Figure 2.21 (see page 52). In this way, DLI is convenient for the interpreta-tion of multimedia document objects, where typically different types of media complement each other to give a richer source of information for a given domain of interest. With the use of a content structure Abox, consistent w.r.t. the MCO ontology, multimedia fusion can be formalized as a logic-based process as well. In the following section we show how content structure information is useful to enable the process of multimedia fusion.

Figure 2.24: DLI results in relation with the document object.

66 CHAPTER 2. DEEP-LEVEL INTERPRETATION OF TEXT