The objective of our implementation is to achieve an interaction between non-linguistic information and syntactic parsing via a single, shared level of semantic representation. So far, our implementation description has covered how WCDG2-external prediction information is propagated into the cross-modally integrated se-mantic representation of sentence meaning. We now outline how WCDG2’s sese-mantic representation interacts with the syntactic representation in the course of parsing.
In WCDG, the representations for dependency structures are level-specific in the sense that each structure uses its own set of edge lables. As a result, the processing on different levels of analysis is informationally encapsulated, i.e., structural changes on a level L1 do not affect the structures on another level L2 unless there is an ex-plicitly defined constraint in the grammar that requires such a correspondence. We hence consider Requirement R12 for the representational encapsulation of represen-tations to be fulfilled by the syntactic and semantic represenrepresen-tations in our model.
The only way that two dependency structures from L1 and L2 can interact with each other is via the structural correspondence rules defined in the interface be-tween those levels of representation. In our model, such a rule typically is a binary constraint relating two edges X and Y such thatX is from L1 and Y is from L2. WCDG2’s syntax-semantics interface establishes correspondence relations between syntactic structural constellations and their semantic correlates. The interface be-tween syntax and semantics enables an immediate and bidirectional interaction be-tween the two representations such that changes in semantic representation directly affect syntactic analysis and vice versa. We consider this a model feature in fulfil-ment of Requirefulfil-ment R4 for representational interfaces.
Our model makes contextual information available at the point in time of syntactic decision making at which they are needed. This way, syntactic candidate structures are assessed for their contextual complianceat the time of their creation rather than subsequently. As we acknowledged previously, WCDG does not provide the capabil-ity of incremental sentence processing yet (cf. Section 4.2.4, p. 83). Requirements R2 and R3 hence cannot be fulfilled in our model implementation. Still, the im-mediate correspondence between syntactic analysis and the contextually-informed semantic representation enables an immediate unidirectional influence of contextual information upon syntactic processing. We consider this a fulfilment of Require-ment R4 which demands the online interaction between non-linguistic modalities and language at parse time.
The syntax-semantics interface propagates the influence of non-linguistic context upon semantics on to syntactic representation via the following mechanism: The integration constraints propagate non-linguistic context information into the seman-tic representation via context integration constraints. Simultaneously, the linguisseman-tic part of semantic analysis interacts with syntactic representation via the correspon-dence constraints in the syntax-semantics interface. The influence of non-linguistic context upon syntactic representation is hence mediated by the semantic repre-sentation as demanded by Requirement R1. In fulfilment of Requirement R13, this mediating effect is achieved by the correspondence rules between the syntactic and semantic representations. Since this correspondence interaction is evaluated at parse time, our model also meets Requirement R14. As there is only one level of semantic representation and this level of representation mediates the meaning-based cross-modal influence of visual context upon syntactic processing, our model meets Requirement R22 for a single and unified semantic representation as well.
With approximately 40 constraints in the syntax-semantics interface spanning more than 400 lines of constraint code1, a comprehensive list of correspondences between the syntactic and semantic levels of analysis is beyond the scope of this work. To provide at least a qualitative impression of the kind of correspondences captured in the syntax-semantics interface, the following list gives a brief overview over some of the more important modelling rules we have implemented.
In an active-voice sentence, the verb’s AGENT is also the subject SUBJ if the verb’s se-mantic valence admits an AGENT.
Conversely, the SUBJ in an active-voice sentence is also the AGENT.
Example: ‘Sie schreibt gerne’
She likes to write.
ADV S
SUBJ
sie schreibt gerne .
AGENT
1This number includes all active lines of code in a constraint. In the absence of a suitable formal measure for the informational complexity of constraint dependency grammars we resort to the purely quantitative measure of expressing the size of the syntax-semantics interface in lines of code. An absolute measure of constraint-dependency grammar expressivity would be desireable for it would permit to compare the informational complexity of different grammars objectively.
In an active-voice sentence, the THEME is the accusative or direct object OBJA if the verb’s semantic valence admits aTHEME.
Conversely, in an active-voice sentence, the verb’s OBJA is also the THEME.
Example:
‘Er isst einen Apfel.’
He is eating an apple.
OBJA
DET S
SUBJ
er isst einen Apfel .
AGENT THEME
In a passive-voice sentence, the verb’s THEME is the subject SUBJ if the verb’s semantic valence admits a THEME. Conversely, in a passive-voice sentence, the THEME is also the SUBJ.
Example:
‘Der Apfel wird gegessen.’
The apple is being eaten.
AUX S
SUBJ
DET
der Apfel wird gegessen .
THEME
In a passive-voice sentence, the verb’s AGENT is the prepositional complement PN in the preposi-tional phrase PP modifying the full-verb if the verb’s seman-tic valence admits an AGENT.
Conversely, in a passive-voice sentence, the PN of a full-verb-modifying PP is the AGENT.
Example:
‘Der Apfel wird von ihm gegessen.’
The apple is being eaten by him.
AUX
PN PP S
SUBJ
DET
der Apfel wird von ihm gegessen .
AGENT
THEME
The verb’s RECIPIENT is its dative or indirect object OBJD if the verb’s semantic valence admits a RECIPIENT. Conversely, the OBJD is the RECIPIENT if the verb’s semantic valence permits.
Example:
‘Sie gab ihm ein Buch.’
She gave him a book.
OBJA
DET OBJD
S
SUBJ
sie gab ihm ein Buch .
AGENT THEME
RECIPIENT
The OWNER of a syntactically modified entity is its genitive modifier GMOD. Conversely, any GMOD is also an OWNER.
Example:
‘Sie haben Kirsas Buch.’
They have Kirsa’s book.
OBJA
GMOD S
SUBJ
sie haben Kirsas Buch .
AGENT THEME
OWNER
In a passive-voice sentence, the verb’s INSTRUMENT is the prepositional complementPN in a full-verb modifying ‘mit’ with or
‘durch’ by prepositional phrase PP. Conversely, in a passive-voice sentence, the PN originates from the INSTRUMENT if the PN is part of a full-verb modifying ‘mit’ or
‘durch’ PP.
Example:
‘Er ¨offnete die T¨ur mit einem Schl¨ussel.’
He opened the door with a key.
PN
DET OBJA PP
DET S
SUBJ
er öffnete die Tür mit einem Schlüssel .
AGENT THEME
INSTRUMENT
In a passive-voice sentence, the COMITATIVE is the prepo-sitional complement PN in a
‘mit’ with prepositional phrase PP that modifies a non-verbal constituent. Conversely, in a passive-voice sentence, the PN of a ‘mit’PP modifying a non-verbal constituent must originate from the COMITATIVE.
Example:
‘Er sieht die Frau mit ihrer Freundin.’
He is seeing the woman with her friend.
PN
DET PP
OBJA
DET S
SUBJ
er sieht die Frau mit ihrer Freundin .
AGENT THEME COMITATIVE
We concede that from a semantic point of view these syntax-semantics correspon-dences are not unduly restrictive. Our experimental results reported in Chapters9, 10, and 11 illustrate, however, that even with these semantically rather loosely cut correspondences very selective syntactic modulations can be effected under cross-modal context integration.
With the use of the extended grammar WCDG2 applies more constraints to its lingu-istic input than WCDG1. One would therefore expect that the quality of analysis in WCDG2 be higher. An aspect counteracting the benefit from the addition of further constraints is that more levels of analysis produce a larger search space. Whether or not the globally optimal dependency structure is found depends on the effectiveness of the frobbing procedure. Guided by the design principle to leave the central pro-cessing mechanisms in WCDG – including frobbing – untouched, WCDG2 operates under these two competing and counteracting influences.
In practice, we find that the majority of the cases in which WCDG2’s syntactic analysis incorrectly deviates from the WCDG1 analysis is due to frobbing not find-ing the correct solution rather than due to incorrect grammar modellfind-ing. The cause for failure to produce the correct syntactic analysis can be tested for in WCDG: A deviant analysis can be modified manually in WCDG. Constraint violations are re-evaluated after every manipulation. If WCDG2 hence scores a manually corrected structure better than the solution found by frobbing, then the obtained structural deviation is due to a search rather than a modelling error. For a more detailed discussion of the extended grammar’s performance on unrestricted input and the challenge of evaluation, see Chapter 8.
From a modelling perspective, the definition of certain syntax-semantics correlations in WCDG2’s grammar presented a significant challenge, in particular when resulting from expressivity limitations in WCDG’s grammar. Limitations encountered were twofold: First, WCDG only permits a maximum constraint arity of two, i.e., a single constraint can only evaluate properties of up to two edges in the dependency tree.
AUX
AUX
AUX S
SUBJ
er hätte gefragt haben können .
AGENT
AUX
AUX
AUX S
SUBJ
er hätte gefragt werden können .
THEME
(a)He could have (have) asked. (b)He could have been asked.
Figure 5.5: Two sentences illustrating that the check for active/passive voice may involve the evaluation of several dependency edges.
Second, the range of features that can be checked for in a constraint is primarily focused on properties of individual edges. Important supra-local properties, such as active and passive voice in a sentence, often manifest themselves on more than one edge and can even span the entire dependency tree.
One constraint in the syntax-semantics interface defines the subject of an agentive active verb in an active-voice sentence to be the verb’s AGENT. In order for this constraint to be evaluated, the global sentence property of active or passive voice needs to be checked for, which may require the evaluation of multi-edge dependency constellations. Figure5.5illustrates an example in which the multi-edge dependency constellations of active and passive voice only differ in a single auxiliary (‘haben’
have versus ‘werden’ been).
In our model implementation’s syntax-semantics interface we overcome this challenge by employing a novel method for capturingglobal andsupra-local sentence properties in WCDG. Our method is based on calls toancillary constraints that check for com-plex edge properties. A comcom-plex property is encoded as a separate constraint. If the property constraints are too complex to be expressed in a single WCDG constraint, ancillary constraints can be constructed that call further ancillary constraints that check for sub-aspects of the complex property and thus effectively decompose the complex property into less complex features that can easily be checked for with WCDG’s standard predicates.
As explained in Section4.2.4, a WCDG constraint can only evaluate the properties of up to two edges as well as their direct neighbours above and below. We extend constraint expressivity by evaluating further ancillary constraints on the neighbour-ing edges and their neighbours. These consecutive calls of ancillary constraints help extend the reach to neighbouring edge properties of the initial constraint by one edge with each ancillary constraint call.
A simple example is to check whether an edge has label AUX and the edge below it also bears the AUX label. This check can still be achieved in a single constraint.
Once the property of the conditions on the edge below get more complex than just checking for its label, we need to encode the check as a separate ancillary constraint,
{X!SYN} : ’AUX above AUX’ : main : 1 : X.label = AUX
& has( X@id, AUX )
;
{X!SYN} : ’AUX above edge above AUX’ : main : 1 : X.label = AUX
& has( X@id, ’Is above AUX’ )
;
{X!SYN} : ’Is above AUX’ : ancillary : 1 : has( X@id, AUX )
;
Figure 5.6: Constraints extending their reach to their immediate edge neighbour (’AUX above AUX’) and to the neighbour’s neighbour (’AUX above edge above AUX’in combination with’Is above AUX’).
e.g.: Check that the edge label is AUX and the edge below it lies above another edge that lies above edge with label AUX. The constraints for these checks are exemplified in Figure 5.6.
One limitation this approach has not been able to overcome is that ancillary con-straints must be unary and can only be applied on the level of analysis of the calling edge. WCDG’s limitation that constraints can only be defined for edges on a maxi-mum of two levels of analysis thus remains. In our model, this did not impose any fundamental modelling restrictions.1
In summary, we can evaluate global sentence properties for trains of contiguous dependency edges on the same level of analysis by consecutive calls to ancillary constraints. More details on our approach of checking global sentence properties with localised constraints are given in McCrae et al. (2008).