7.2 Evaluation Design
7.2.2 The Algorithm of Barzilay & McKeown, 2005
To date, the algorithm ofBarzilay & McKeown (2005) is the first and the only example of an implemented and evaluated sentence fusion system. Therefore, we reimplemented and adapted it to process German data to have a point of comparison for deFuser. In this section we first describe the architecture of the original fusion system and then introduce our reimplementation for German.
Theme (sentence group) construction: The sentence grouping or theme construction com-ponent, SimFinder, used byBarzilay & McKeown (2005) is more complex than ours. It uti-lizes supervised learning and is trained on a set of linguistic features – e.g., WordNet synsets, syntactic dependencies. However, a simpler algorithm ofNelken & Shieber (2006) which our sentence grouping module draws upon outperforms SimFinder (see Chapter3).
Theme (group) ranking: Sentence groups are ranked, and the n top-scoring ones are chosen for fusion, i.e., from those n groups n new sentences are later generated. Such factors as the size of the group, the similarity within the group, and the number of lexical chains (Barzilay
& Elhadad, 1999) running through the group contribute to the rank. The groups are ordered chronologically, i.e., themes appearing in earlier news articles precede those which appeared later.
Tree transformation: Dependency trees are obtained from phrase structure parses output by Collins’s 2003 parser with a set of rules. These are further transformed in several re-spects: grammatical features and auxiliary nodes are removed from the tree and recorded;
noun phrases are flattened. Two examples of transformed trees corresponding to the sentences in (7.1-7.2) (taken fromBarzilay & McKeown (2005)) are given in Figure7.3.
(7.1) IDF Spokeswoman did not confirm this, but said the Palestinians fired an antitank missile at a bulldozer.
(7.2) The clash erupted when Palestinian militants fired machine guns and antitank missiles at a bulldozer that was building an embankment in the area to better protect Israeli forces.
spokeswoman IDF
antitank missile
say fire Palestinian
bulldozer at confirm
this but
(a) Transformed dependency tree of (7.1)
militant Palestinian
machine gun antitank missileand
build in
area force
Israeli embankment
better protect that
fire
at bulldozer when
clash erupt
(b) Transformed dependency tree of (7.2)
Figure 7.3: Transformed trees of sentences (7.1-7.2)
(7.3) The army expressed “regret at the loss of innocent lives” but a senior commander said troops had shot in self-defense after being fired at while using bulldozers to build a new embankment at an army base in the area.
Tree alignment and basis tree selection: Pairs of transformed trees from one group are aligned, and each alignment gets its alignment score. The alignment algorithm proceeds in a bottom-up manner and, using dynamic programming, finds locally optimal alignments by taking node similarity as well as structural similarity into account. Figure7.4shows the align-ment structure of the trees in Figure7.3. Solid lines represent aligned edges; dotted and dashed lines represent unaligned edges from the trees in Figures7.3aand7.3brespectively. Note that the nodes corresponding to Palestinian and Palestinian militants are aligned although they are not identical. We skip detailed presentation of the algorithm as well as its pseudocode which can be found in the cited article. Here, it is important to stress the following features of the algorithm:
• Several sources of information are used to measure node similarity: lemma identity, lexical relations encoded in WordNet (Fellbaum, 1998) and an automatically extracted paraphrase dictionary.
• In some cases structural similarity enforces alignment of nodes which are neither syn-onymous nor paraphrases. For example, chances are high that two verbs are aligned
7.2 Evaluation Design 119
spokeswoman IDF
fire
at bulldozer when
clash erupt
(machine gun antitank missileand) Palestinian
(militant)
confirm
this but say
build in
area force
Israeli embankment
better protect that
Figure 7.4: Alignment structure of the trees in Figures7.3a-7.3b
given that their subjects and objects are aligned.
• The algorithm finds a locally optimal alignment – searching for a globally optimal one is NP-hard.
• Once all pairwise alignments and their respective scores have been found, the centroid tree of the group is identified. This tree, called basis tree, has the maximum similarity to other trees in the group.
Basis tree augmentation and pruning: During this stage the basis tree is modified in two ways. Firstly, alternative verbalizations are added. These are taken from the nodes which have been aligned with the nodes in the basis tree. Furthermore, a subtree from a tree other than the basis one is inserted provided that its root is aligned with a node of the basis tree and that the subtree appears in at least half of the sentences from the group. This rule is quite restrictive but reduces the chances of generating ungrammatical or semantically unacceptable sentences. Secondly, certain subtrees are removed provided that they are unaligned. The list of prunable components includes a clause in the clause conjunction, relative clauses, adverbs and prepositional phrases. The transformed tree in Figure7.3bis the basis tree of the sentence group (7.1-7.3). Figures7.5aand7.5brepresent the basis tree after augmentation and pruning respectively.
fire
at bulldozer when
clash erupt
(machine gun antitank missileand) Palestinian
(militant)
build in
area force
Israeli better
protect that
(new) embankment
to while
using
(a) Augmented basis tree
(machine gun antitank missileand) Palestinian
(militant)
fire
at
while using
build that
in area bulldozer
to
(new) embankment
(b) Basis tree after pruning
Figure 7.5: Basis tree from Fig.7.3bafter augmentation and pruning
Linearization: The best linearization of the dependency structure is found by overgenera-tion and ranking. From the set of all possible strings the one with the lowest length-normalized entropy is selected. The entropy is estimated with a trigram language model trained on a col-lection of 60M news. The linearizations differ not only in the set of words they cover and word order but also in node realizations (e.g., new embankment vs. embankment). Informa-tion available from the input has been used to reduce the number of possible strings (e.g., the fact that two words are found in a certain order in all the input sentences). Furthermore, the number of linearizations is limited to the first 20,000. The best linearization of the dependency structure in Figure7.5bis reported to be Palestinians fired an antitank missile at a bulldozer.