7.4 The Implemented Grammar
7.4.2 Effects and Side Effects
al-lowed to contain gaps; this rules out ill-formed fragments like “the father of”, which otherwise would be allowed (as an NP containing a gap where the complement of the preposition is).
FRAGH-DTR sign & [SYNSEM.NON-LOCAL.SLASH 0-dlist], FRAGNH-DTR sign
].
The types for the three dimensions now are direct daughters of this general type:
frg_md :< frg.
frg_arg :< frg.
frg_mood :< frg.
The type frg md (for fragment-modification) has two daughters, one for fragments that are modified by an adverb and one for those that aren’t. Modification works basically as described above in (286) in the previous section, with some feature names being different. frg arg is a more diverse hierarchy, with sub-types for altogether six different arguments categories (NPs, lexical and functional PPs, VP[inf ], VP[bse] and S[comp]; we do not provide analyses for the other types mentioned above); again the con-tribution of the subtypes is as shown in the previous section, in Section 7.3.2. The type frg mood then finally has only two daughters, one for interrogative and one for declarative fragments. As the reader will have noticed, this leaves out imperative fragments; moreover, we make the restrictive assumption here that only wh-phrases are interrogative fragments. Both restrictions are made for the same reason: since obviously no information about verb-inversion for example is available, the only remaining information source for distinguishing for example “Peter.” from “Peter?” or “Peter!” would be interpunction (or, in the case of a speech recognition grammar, intonation). The parsing system we use, however, does not (at the time of writing) exploit this information, and so to avoid multiplying parses (by giving three ana-lyses for e.g. “Peter”) we have decided for this restrictive approach. This, however, is not a conceptual necessity and could be changed if needed.
All in all, this results in 18 different construction types as the nodes of this hierarchy, which through multiple inheritance can be specified very parsimoniously, as the example for non-modified interrogative NP-fragments shows:
np_nm_i_fragment := np_fragment & frag_int & fragment_nm.
be too costly. This is what we discuss here for the ERGwith our modifications for fragments (in the following, ERGFRAG). We provide three sets of evaluations, all done with[incr tsdb()](Oepen &
Flickinger 1998): first, the coverage of theERGFRAG with respect to the phenomenon it is supposed to analyse; second the (side-)effects of the modifications on analyses of full sentences; and lastly the gain in coverage of real-world utterances.
Note that these evaluations, especially those with larger data sets, are only intended to show tendencies or trends in behaviour, since the main indicator used here is simply the number of parses the grammar offers for a given string. Ultimately, a grammar should be evaluated by checking whether the contextu-ally appropriate parse for a particular item is in the set of those given by the grammar. We have done this only for a small set of data, since it is very time consuming; and so our evaluation overall should only be taken as indicating a tendency.
Coverage of Phenomenon
To help with the construction of the grammar, early on we devised a very small test-suite of fragment examples for use with[incr tsdb()](printed in Appendix A.2); this allowed us to judge the effects of changes to the grammar with respect to analyses of the phenomenon. For the evaluation of the finished grammar, however, this test-suite of course isn’t that useful: in a sense, it constitutes our ‘training data’, and so for types of fragments we decided to provide with a representation,13 our grammar reaches the ideal coverage of 100% of the test-items. The results are summarised in Table 7.1.
What is interesting here, however, is the ambiguity the grammar exhibits, as indicated by the numbers in the fifth column of the table. For example, VP-fragments have on average more than two parses, whereas modified fragments have on average more than five parses. To get an idea of these parses, let’s go through some examples. First, for the string “kick Sandy”. This string gets two parses from the
ERGFRAG(we forgo showing them here), only one of which is as a fragment. The other one, naturally, is as an imperative sentence. This makes clear that the classification of the items in the test-suite into types of fragments is not exclusive: being a fragment is not a syntactic property, and hence it is possible that there are other parses for a given string that could be a fragment. All we are interested in here is the ambiguity our grammar adds, and for this example we fared well (since we only added exactly one reading). A bit more surprising perhaps are some of the parses for “to kick Sandy”. (295) first shows the parse that corresponds to the analysis we have given for this string so far: a VP[inf ] that is projected to S.
(295)
13We should stress again here that there was no substantial reason for not extending our approach to these types; we simply concentrated on the major types. Of course, the impact of providing analyses for the types we left-out would have to be judged again.
total positive word lexical parser total overall Phenomenon items items string items analyses results coverage
] ] φ φ φ ] %
NP frag 5 5 2.40 6.60 2.40 5 100.0
PP frag 3 3 2.00 4.33 2.00 3 100.0
VP frag 7 3 3.00 8.67 2.67 3 100.0
ADV frag 3 3 1.00 1.67 0.00 0 0.0
ADJ frag 2 2 1.00 2.00 0.00 0 0.0
Int frag 8 8 1.37 4.50 1.50 8 100.0
Slashed frag 6 0 0.00 0.00 0.00 0 0.0
Modified frag 3 3 4.00 11.00 5.33 3 100.0
Conj frag 5 5 3.00 11.60 3.33 3 60.0
Total 42 32 2.19 6.50 2.56 25 78.1
(generated by [incr tsdb()] at 27-nov-2002 (15:38 h))
Table 7.1: Coverage of Fragment-Test-Suite
to COMP
kick STEM VP-NF
sandy STEM
N NP VP-NF VP
S
The next two parses, shown in (296), are less obvious. In the first, “to kick Sandy” is interpreted as a PP meaning “in order to kick Sandy”—a parse we didn’t think about before, but that seems correct.
The second parse interprets the string as consisting of such a PP, modifying a sentence. A full sentence instance of this construction is “to sing, Peter came here”; with a fragment as sentence from which the PP is extraposed this surely sounds rather unplausible, but there is no principled reason for ruling this out. In any case, the discussion of this example nicely demonstrates the advantages of using a wide-scope grammar for implementation, since the interaction with such a construction would otherwise not have been noticed.
(296)
to P-S
kick STEM VP-NF
sandy STEM
N NP VP-NF PP-S
S
to P-S
kick STEM VP-NF PP-S
sandy STEM
N NP
S S
Limitations of space preclude us from discussing many more examples, and so we only show here one more, to explain why modified fragments have so many more parses. The fragment “probably to kick Sandy.” receives six analyses, three of which are like the parses shown before for “to kick Sandy”, with the sentence modified by the sentence-adverb reading of “probably”. The remaining three are shown in (297). Two of these are the result of application of the ‘S→ADV XP’ rule; in the remaining one the ADV is part of the VP, and the resulting VP is projected to S.14
(297)
probably ADV-PR
to P-S
kick STEM
V
sandy STEM N NP VP-NF PP-S S
probably ADV-PR
to COMP
kick STEM
V
sandy STEM N NP VP-NF VP S
probably ADV-PR
to COMP
kick STEM
V
sandy STEM N NP VP-NF VP VP
S
We now come to the side-effects of having rules for fragments on the full-sentences grammar.
Side-Effects
To test for possible adverse effects on the analyses of full-sentences, we also ran batch-parses of a test-suite of full sentences, namely theCSLI-test-suite which is distributed with[incr tsdb()]. This test suite consists of 1348 sentences, of which 961 are marked as syntactically well-formed and 387 as
ill-14This makes us expect that the following structure is licensed, which indeed it is: ‘[Sprobably [S[f rag]probably [V Pto [V P
probably kick sandy]]]]’. Again, there is no principled syntactic reason to exclude such a parse.
formed. Table 7.2 shows a comparison of the originalERGwith four versions of theFRAG-ERG,15with respect to the average number of parses per sentence, the percentage of sentences that are marked as well-formed which do get at least one parse (coverage), and the percentage of ill-formed sentences that nevertheless get a parse (overgeneration).
Version of Grammar Average # parses % Coverage % Overgeneration
LinGO ERG, 20/11/02 2.86 81.4 33.9
ERG+frag - - 3.23 82.3 36.4
ERG+frag +- 3.33 82.3 36.4
ERG+frag -+ 3.59 82.3 37.7
ERG+frag ++ 3.69 82.5 37.7
Table 7.2: Competence comparison of the originalERGwith four versions of the fragment-ERG
As these data show, the rules for fragments introduce new ambiguity into the full-sentence part of the grammar; only slightly so for the most restricted version ofERG-FRAG(which only adds rules for NP-and PP-fragments NP-and does not allow embedding), up to on average almost one more parse for the version with VP-rules and embedding. We also note a slight increase in coverage (i.e., sentences known to be well-formed that do get parsed) and a more pronounced increase in overgeneration (ill-formed sentences that nevertheless do receive a parse).
But what does this tell us? One has to be careful when interpreting these numbers. It seems a reasonable conclusion to say that the addition of fragment-rules does not lead to an explosion of readings that would render the grammar practically unusable. Without further analysis, however, this does not tell us anything about whether the additional readings (of what are meant to be full sentences) are erroneous or not. The problem of course again is that ‘fragmenthood’ is not a syntactic criterion, and so some strings that can be analysed as sentences can also be analysed as fragments (and hence an increase in ambiguity would be unavoidable); we have just discussed such an example in the previous section (“kick Sandy”).
And by the same token, even the increase in coverage and the increase in ‘over-generation’ need not be due to unwanted side-effects, since there could be items that are now parsed quite legitimately as fragments.
Hence, a more detailed analysis is required. To begin with a simple example, (298) shows a parse for the string “the dog barks” which is added by our fragment rules:
15The two varying factors are whether VP-fragments are analysed or not and whether embedding of fragments is allowed or not;
hence ‘ERG+frag -+’ denotes a version of the fragment grammar where VP-fragments are excluded, but embedding is allowed.
(298)
the DET
dog STEM
N
STEM barks
N N N NP
S
It turns out that the grammar lists ‘bark’ as a noun as well as a verb, and so ‘dog barks’ forms a compound noun which is then part of an NP-fragment. So in this case, the additional parse seems to be legitimate.
A more involved example—which occurs in the test-suite used to get the data quoted above—is the sentence “Abrams showed the office to Brown.” The unmodifiedERGgives four readings for this, to whichERGFRAG adds six further readings. Five of these additional readings are variations of the one shown on the left in (299); in all of them ‘show’ takes a sentence as complement (as in ‘Peter showed that the Goldbach conjecture is true.’), a sentence which however is a fragment. In the parse shown on the left in (299), this fragment is ‘the office to Brown’ parsed as an NP; there are also parses where
‘the office’ is the direct object of show and ‘to Brown’ as a PP-fragment is the sentential complement.
These readings are certainly unwanted, and point to a problem of the version of the fragment grammar that allows embedding, namely that it is too unrestricted. Ideally, fragments should be marked as a special kind of sentence only certain kinds of verbs select for (e.g. ‘think, say, believe’, etc.). To do so, however, would require further analysis of where embedding of fragments is possible, and also would require changes to the sentence grammar, which in this first step we did not want to change. So we conclude here that allowing embedded fragments systematically overgenerates.