PILOT STUDY: PARAPHRASE ACQUSITION 107 Configurations

Aligner+ LCS+ Word+ LCS+Word+

Phrase Extraction Chunk N-Gram Chunk Our Corpus

Paraphrase 15% 36% 32%

Related 21% 26% 21%

Sum 36% 62% 53%

The msr Corpus

Paraphrase 38% 44% 49%

Related 20% 19% 18%

Sum 58% 63% 67%

Table 5.7: Distribution of the extracted fragment pairs of our corpus and msr corpus.

5.6.4 Discussion

Table 5.8 shows some examples from the best two settings. From our corpus, both simple paraphrases (“Governor ... said” and “Gov. ... an-nounced”) and more varied ones (“rose to fame as” and “the highlight of his career”) can be extracted. It is clear that the smoothing and extrac-tion algorithms do help with finding non-trivial paraphrases (shown in Figure 5.2). The extracted phrase “campaign was” shows the disadvan-tage of n-gram-based phrase extraction method, since the boundary of the fragment is improper. Using a chunker can effectively exclude such problems, as shown in the lower part of the table, where all the extracted paraphrases are grammatical phrases. Even from a parallel paraphrase corpus at the sentence level, the acquired fragment pairs (w/o context) may be non-paraphrases. For instance, the second pair from themsr cor-pus shows that one news agency gives more detailed information about the launching site than the other, and the last example is also debatable, whether it’s “under $200” or “around $200” depending on the reliability of the information source.

As far as we know, Munteanu and Marcu (2006)’s bilingual fragment extraction method has not yet been applied to the task of monolingual paraphrase extraction. Zhao et al. (2008) extracted paraphrase fragment

From Our Corpus: word aligner + n-gram-based phrase In San Juan, Puerto Rico, Governor Pedro Rosello said

Paraphrase the storm could hit the US territory by Friday, ...

InPuerto Rico, Gov. Pedro Rossello announced that banks will be open only until 11 a.m. Friday and ...

Kunstler rose to fame as the lead attorney for ...

Paraphrase The highlight of his careercame when he defended ...

... initiatedthe air attacks in response to Serb shelling of

Invalid Sarajevo that killed 38 people Monday.

The campaign wasto respond to a shelling of Sarajevo Monday that killed 38 people.

From msr Corpus: LCS + word aligner + chunk-based phrase ... Jordan Green,declined to comment.

Paraphrase ... the prelate’s private lawyer, said hehad no comment.

... to blast off between next Wednesday and Friday

Related from a launching site in the Gobi Desert.

... to blast off as early as tomorrow or as late as Friday from the Jiuquan launching sitein the Gobi Desert.

... Super Wireless Media Router, which will be available

Related in the first quarter of 2004,at under $200.

The router will be available in the first quarter of 2004 and will costaround $200, the company said.

Table 5.8: Some examples of the extracted paraphrase fragment pairs.

5.7. SUMMARY 109 pairs from bilingual parallel corpora, and their log-linar model outper-forms Bannard and Callison-Burch (2005)’s maximum likelihood estima-tion method with 67% to 60%. Notice that our starting corpora are (noisy) comparable corpora instead of parallel ones, and the approach is almost unsupervised, so that it can be easily scaled up to other larger corpora, e.g., news websites.

In sum, we presented our work on paraphrase fragment pair extrac-tion from monolingual comparable corpora, inspired by Munteanu and Marcu (2006)’s bilingual method. We evaluated our intermediate results at each of the stages using MTurk. Both the quality and the quantity of the collected paraphrase fragment pairs are promising given the minimal supervision. As for the ongoing work, we are currently expanding our extraction process to the whole gigaword corpus, and we plan to apply it to other comparable corpora as well.

For future work, we consider incorporating more linguistic constraints, e.g., using a syntactic parser (Callison-Burch, 2008), to further improve the quality of the collection. More importantly, applying the collected paraphrase fragment pairs to other NLP applications will give us a better view of the utility of this resource. As Bosma and Callison-Burch (2007) have utilized similar resources in the RTE task, a better way of using such paraphrase fragment pairs is still under exploration.

5.7 Summary

To sum up, we identify important issues encountered in using inference rules for textual entailment and propose methods to solve them. We explore the possibility of combining a collection obtained in a statistical and unsupervised manner, DIRT, with a hand-crafted lexical resource in order to increase the usefulness of inference rules for applications. We also investigate ways of effectively applying these rules. The experimental results show that although coverage is still not satisfactory, results in terms of precision are promising. Therefore, our method has the potential to be successfully integrated in the extensible framework described in Chapter 3 as a specialized module.

The error analysis points out several possible future directions. The tree skeleton representation we use needs to be enhanced in order to capture the relevant fragments of the text more accurately. A different issue remains the fact that a lot of rules we need for textual entailment

detection are still missing. Further investigations into the limitations of the DH as well as a classification of the knowledge we want to encode into the inference rules would be a step forward towards solving this problem.

At last, we present a pilot study of acquiring paraphrased fragment pairs using monolingual comparable corpora, which can be viewed as an alternative to the DIRT-style rules acquired based on the DH.