entities in the visual scene or cases of visual ambiguity, such as in a snapshot of a dynamic, potentially bi-directional event which makes it impossible to tell in which direction the scene is evolving. It would be cognitively highly ineffective if humans always integrated visual information with the same strength at all times. More to the point, the degree to which humans rely on visual information to support their linguistic processing is dynamic and adjusts situation-specifically. With the introduction of the modelling parameter context compliance, we have incorporated precisely this aspect as an important feature in our model.
Syntactic Attachment Modulation by Soft Integration
Experiment 2 in the preceding chapter showed that the CIA successfully performs the integration of propositional semantic context information into the process of syntactic parsing. The hard integration scenario discussed effectively models an absolute dominance of the visual modality over linguistic processing. In most real-world situations, however, visual understanding is subject to challenges such as uncertainty, conflicting information or perceptual ambiguity. It is therefore implau-sible to assume that the integration of visual information into linguistic processing is always performed with the same strength. More realistically, humans dynamically adjust the strength with which they integrate the semantic information from visual context into linguistic processing. In some cases visual information will have a strong effect upon linguistic processing while in other cases it will remain inconsequential.
The ability to perform dynamic adjustments of integration strength can suitably be modelled in the CIA based on the WCDG’s capability to process weighted con-straints. In this chapter, we investigate the results of context integration via soft constraints as a viable alternative to the previously studied hard integration. Our experimental findings show that soft integration provides a number of benefits over hard integration such as context integration without a damage to contextually unre-lated syntactic and semantic dependencies, the accommodation of conflicting visual and linguistic information in a uniform linguistic representation as well as diagnostic capabilities to highlight semantic dependencies in discord with the modelled con-textual information. The capability of a cognitive system to perform diagnosis of which parts of a given input violate context-driven expectations is highly impor-tant in contextualised cognition. Rather than just to say that a given sentence is inconsistent with contextual expectations diagnosis permits to say which aspects of linguistic analysis are in conflict with context-based expectations. In natural sys-tems, the ability to perform such diagnostics enables a more specific and effective response to and interaction with the environment.
163
10.1 Experimental Motivation
The detailed analysis of the experimental findings in Experiment 2 reported in Section 9.5 revealed that hard context integration can have the undesirable side effect of modulating semantic dependencies in referentially unrelated parts of the input sentence that should remain unaffected by the give visual context information.
Concretely, we observed that the semantic analysis of the main clause, for which no context representation was available, was modulated by the integration of visual context information related to the subclause.
While the system of constraint weights is robust enough to leave the syntactic analysis unaffected, the hard integration constraints on the semantic levels of analysis enforce partial defects on the semantic analysis of the introductory main clause. As a result, all hard-integration structures were missing a specific semantic dependency that should have been present.
It would be desirable to achieve context integration that affects only those areas of the sentence that the visual context actually refers to. Ideally, context inte-gration should be selective such as to leave all other aspects of linguistic analysis unchanged. For this reason, soft integration is an attractive option: It permits to adjust the strength with which contextual influences are enforced upon linguistic analysis.
In Experiment 3 it is our aim to find an appropriate weighting for the strength of context integration: On the one hand, visual context integration should be strong enough to drive linguistic analysis in line with the visual scene information; on the other hand, context integration must be soft enough such as not to enforce linguistic structures that violate any of the harder constraints in the grammar. An example for one of these constraints is the semantic valence constraint with a constraint weight of 0.1 which was violated under hard integration by the absence of the AGENT de-pendency in the context-integrated structures (see Section9.5).
With soft context integration we expect the hypothesis space to take the same size as for empty context integration. This is because visual context does not impose any hard constraints on the linguistic analysis anymore. The violation of the integration constraints now does not result in the exclusion of a structural candidate from the hypothesis space anymore. The hypothesis space hence contains more structural candidates and it should also be easier for frobbing to progress towards the opti-mal solution. We therefore expect processing times and the number of constraint evaluations to go down for the integration of non-empty context models.