A Bag-of-Words Approach

Roughly thinking, the more overlapping words there are between T and H, the more possible that T entails H, and vice versa. Like the following two examples,

Dataset=RTE-2 Dev Id=47 Task=IR Entailment=YES

Text: Women form half the population and 54% of the voters in the country, yet are very poorly represented in parliament.

Hypothesis: Women are poorly represented in parliament.

Example 14

Dataset=RTE-2 Dev Id=140 Task=IR Entailment=NO

Text: Aspirin, an inexpensive drug helps protect survivors of heart attack and stroke from subsequent heart attacks and death, and even helps reduce the number of deaths that occur within the first hours following a heart attack.

Hypothesis: People experienced adverse effects while taking aspirin.

Example 15

In the first example, every word in H also appears in T, and the meaning of H is just what one part of T conveys; while in the second example, “aspirin” is the only overlapping word between T and H, and T and H convey irrelevant information. Therefore, it seems that the number of overlapping words can tell us whether T entails H.

However, our rough assumption is not always true. See the following two examples,

Dataset = RTE2-dev Id=103 Task=IR Entailment=YES

Text: This paper describes American alcohol use, the temperance movement, Prohibition, and the War on Drugs and explains how legalizing drugs would reduce crime and public health problems.

Hypothesis: Drug legalization has benefits.

Example 16

Dataset=RTE2-dev Id=35 Task=IE Entailment=NO

Text: Meanwhile, in an exclusive interview with a TIME journalist, the first one-on-one session given to a Western print publication since his election as president of Iran earlier this year, Ahmadinejad attacked the "threat" to bring the issue of Iran's nuclear activity to the UN Security Council by the US, France, Britain and Germany.

Hypothesis: Ahmadinejad attacked the UN Security Council.

Example 17

In the first T-H pair, only half of the words appear both in T and H, but the entailment relationship holds. Here, “reduce crime and public health problems” entails “has benefits”. In the second one, every word of H can be found in T, but fortunately, the answer is “NO”.

“Ahmadinejad” didn’t attack “the UN Security Council” but “the ‘threat’ to bring the issue … to the UN Security Council”. The first one is rather difficult for the moment; therefore we will start handling the second one in the next subchapter (3.3).

Before that, one point should be mentioned here. BoW method has its advantages. There are some examples as follows seem very difficult to manage even for human beings, but using BoW method can predict the answer correctly without any “deep thinking”,

Dataset=RTE2-dev Id=513 Task=IE Entailment=YES

Text: These acoustic methods are now expected to be useful for the long-range remote sensing of schools of fish as well as for distant ocean bottom characterizations.

Hypothesis: Ocean remote sensing is developed.

Example 18

Dataset=RTE2-test Id=155 Task=QA Entailment=YES

Text: The EZLN differs from most revolutionary groups by having stopped military action after the initial uprising in the first two weeks of 1994.

Hypothesis: EZLN is a revolutionary group.

Example 19

If the system really needs to obtain the answer after understanding both the T and H, as human beings do, it must have a deep semantic parser to know “some methods are expected to be useful for some advanced technique” entails “some not so advanced technique has already been developed.” Therefore, here, the acoustic methods are expected to be useful for long-range remote sensing of schools of fish, implies that remote sensing is developed.

Furthermore, the second usage of the methods, ocean bottom characterizations, implies the remote sensing is also in the ocean. Altogether, ocean remote sensing is developed. Example 19 is another difficult example, which asks the systems to know “EZLN is a revolutionary group”, if it can be compared with other “revolutionary groups”.

How does our BoW method deal with these pair? It is straightforward. It does not care about the relationship between “ocean” and “remote sensing”, which are just overlapping words between T and H; neither does it care about the implicature between expected usage for some technique and the development state of that technique. It views these pairs as “most of the words are the same.” More examples are given below,

Dataset=RTE3-dev Id=216 Task=IR Entailment=YES Length=short Text: Anti-nuclear protesters on Wednesday delayed the progress of a shipment of radioactive waste toward a dump in northern Germany. The train stopped for the fourth time since crossing into Germany as it neared the northern town of Lueneburg.

Hypothesis: Nuclear waste transport delayed in Germany.

Example 20

Dataset=RTE3-dev Id=730 Task=SUM Entailment=YES Length=short Text: The IAEA board in February referred Iran to the Security Council, suggesting it had breached the Nuclear Nonproliferation Treaty and might be trying to make nuclear weapons.

Hypothesis: Iran might be trying to make nuclear weapons according to the IAEA board.

Example 21

In the first example, all the words in H are distributed in the first sentence of T, which is difficult to resume the same relationships between these words, but the method still works well. The second example is even more difficult, because anaphora resolution is needed to know the subject of “might be trying to make nuclear weapons” is “Iran” in T. It can also be handled by the BoW method simply due to the high word overlapping ratio between T and H.

As a summary, the examples in this subchapter give us some basic ideas to handle the RTE task. It seems that word overlapping itself is not enough to cover all the cases, though it can predict some T-H pairs. However, notice that, word overlapping calculation can be done on any pair, which means every pair has a word overlapping ratio between H and T. This characteristic makes this Bag-of-Word approach extremely robust, ignoring the accuracy.

That’s why we use it as a backup strategy, which will deal with all the pairs cannot be solved by our main approach.